Unified Radix Cache in SGLang: one tree for hybrid model prefix caching

Adapted from the LMSYS article on Unified Radix Cache.

Background

Prefix caching reuses KV when requests share a token prefix. Under full attention, a later request can reuse the cached prefix and compute only the new tokens.

Hybrid models combine three kinds of data with different reuse rules: FULL (full attention) needs KV along the matched prefix, SWA (sliding window attention) needs a continuous window immediately before the reuse boundary, and MAMBA needs a snapshot of the recurrent state, or checkpoint, at that boundary.

DeepSeek-V4 pairs FULL with SWA, Kimi-K3 pairs FULL with MAMBA for its KDA recurrent state, and Inkling uses all three. SGLang handles these combinations on one shared tree, with each component defining its own reuse rule.

Unified Radix Cache

Design and mechanism

Each radix node stores a token span and a slot for each component. FULL and SWA slots hold KV page indices; MAMBA holds a checkpoint at the prefix endpoint. SWA data outside the required window can stay cached or be evicted independently, leaving an empty slot called a tombstone.

Components handle their own splits, inserts, locks, and evictions. Reclaiming part of a node’s SWA data starts by splitting the node at the retention boundary. A MAMBA checkpoint stays at its original prefix endpoint; a request reusing it obtains a private copy through copy-on-write, then continues updating its state.

The safe reuse boundary

Matching walks the matched path, and each component checks its data at every candidate boundary. Each candidate is evaluated independently. Computation resumes from the deepest boundary accepted by all components.

HiCache across memory tiers

HiCache extends Unified Radix Cache across GPU L1, host memory L2, and external storage L3. See HiCache in SGLang for its tiering and data transfers.

Components and sidecars

DeepSeek-V4’s FULL and SWA use separate page indices, mapped by the allocator. Auxiliary pools for compressed KV, indexer buffers, and compressor states act as sidecars: they share their component’s indices and transfers. Three follow FULL and two follow SWA.

Multi-turn benchmark

Multi-turn conversations keep extending the reusable prefix. A comparison of L1 only, L1+L2, and an additional 500 GiB Mooncake L3 tier shows DeepSeek-V4-Flash reaching 145.5K effective input tokens/s with L3, up from 9.4K with L1 only, with average TTFT below 9 seconds.

Session-aware eviction

Applications use a stable session_id to identify requests from the same conversation. SGLang tracks that session’s references to cached entries. Calling /close_session removes its references; shared prefixes keep those from other sessions.

FULL orders unlocked, eligible entries by reference presence, reference count, and then the base eviction policy, usually reclaiming unreferenced data first. Session retention applies to L1/L2; the storage backend manages L3 eviction.

SWE-bench results

On SWE-bench, enabling Unified Radix Cache together with session-aware eviction lowers TTFT by up to 11.0% on DeepSeek-V4-Pro and 16.6% on Qwen3.5-397B-A17B, compared with HiRadixCache and LRU.

Rust tree core

SGLang runs tree traversal, lock bookkeeping, and LRU updates in Rust to reduce their CPU overhead on the scheduler. Python handles physical KV allocation and cache orchestration, applying the operations Rust returns to the pools.

The L1-only Rust prototype #29074 reduced TTFT by 38% for SWA, 10% for full attention, and 5% for hybrid SSM against the Python tree in a 200-turn conversation benchmark.

The subsequently merged Rust TreeCore #32710 supports FULL, SWA, and MAMBA combinations and HiCache. Session-aware caching continues to use the Python tree core.