HiCache in SGLang: hybrid-model caching with up to 15.5× input throughput
SGLang reuses cached prefixes to avoid repeated prefill, but GPU memory cannot retain every growing conversation. This article explains SGLang HiCache and how it extends the cache to CPU memory and external storage. In a DeepSeek-V4-Flash multi-turn benchmark, adding Mooncake L3 brings effective input throughput to 15.5× the GPU-only baseline, with cache hits near 98%.
Background
A model computes keys and values, or KV, for the tokens it reads. During generation, attention uses this KV to look back at the context. When another request starts with the same token prefix, SGLang can reuse its KV and compute only the new suffix. This reduces prefill work and time to first token, or TTFT.
The capacity of GPU memory limits that reuse. Conversations grow with every turn, and more clients bring more histories. Once the cache fills up, a prefix may be evicted just before the next request needs it. SGLang’s HiCache retains these prefixes in larger memory tiers. For that capacity to help, restoring a prefix must cost less than recomputing it, and the restored state must be sufficient to continue the model from that prefix.
HiCache first added CPU memory, then external storage. Later work extended recovery to hybrid-model and draft state, and introduced a proposal to use host memory only as a transfer buffer. The timeline places these changes in context; the rest of the article follows the engineering problems they address.
HiCache
Design and mechanism
Keeping every reusable prefix on the GPU would compete with the memory needed for active requests. HiCache therefore separates where a prefix is retained from where the model uses it. GPU HBM is L1, CPU DRAM is L2, and external storage is L3. The GPU still performs attention; lower tiers preserve data that would otherwise be discarded. A cache controller backs up and reloads that data.
Suppose two requests start with the same document but ask different questions. SGLang can reuse the document’s KV and compute only the new question. HiRadixTree matches the shared prefix and tracks its local cache locations; HiCache brings lower-tier copies back to the GPU when needed.
Cache when the request arrives
An L1 hit is ready to reuse. An L2 hit must be loaded into L1. An L3 hit must first be fetched into L2, then loaded into L1. Only the parts missing from all three tiers need recomputation. This is HiCache’s benefit: a prefix evicted from GPU memory can still avoid recomputation if a lower tier retains it. Reuse requires identical prefix tokens; similar meaning alone is not enough.
Engineering overview
The hierarchy creates a path for retaining and recovering prefixes. Making that path useful introduces several implementation problems:
- Memory layout: By default, GPU computation groups data by model layer, while storage I/O groups it by page. How can transfers between these layouts remain efficient?
- Overlapping data transfers and computation: which transfers can overlap with GPU computation to keep the GPU from waiting?
- Cache policies: prefetching and backup spend time, bandwidth, and capacity. When is waiting or saving another copy worthwhile?
- Hybrid-model and draft state: a token match can leave required history missing. What must travel with the prefix for recovery to be complete?
- Host memory: each instance has a private L2 pool. When L3 retains the history, how much data must remain in host memory?
- Storage backends: different systems expose different I/O implementations. How can they share HiCache’s scheduling and recovery mechanisms?
Memory layout
This optimization handles the different KV layouts used by GPU computation and storage I/O. The GPU processes one model layer at a time, so by default its KV is grouped by layer for convenient access. External L3 storage reads and writes pages: for example, page 1 contains tokens 1–4 and page 2 contains tokens 5–8.
If Host keeps the same layout, transfers incur the overhead of many small copies, as shown in the diagram. With a page-first Host layout, a whole page can be transferred to external L3 storage at once, reducing the number of transfers and their overhead.
HiCache therefore groups Host KV by page for transfers. It converts the layout during transfer, as shown in the diagram. Page 1 is highlighted: before, its pieces occupy separate Host regions; after, they sit together. Host → storage backup can read that contiguous region directly, and storage → Host recovery can fill it directly, reducing overhead.
On GPU ↔ Host, SGLang’s GPU-assisted I/O kernels achieve up to 3× the transfer throughput of the standard cudaMemcpyAsync path. On Host ↔ storage, page-first layout enables large contiguous reads and writes; zero-copy lets the storage client access those blocks directly in the Host pool. Together, page-first and zero-copy deliver up to 2× the Host–storage throughput.
Overlapping data transfers and computation
The previous section explained how to make transfers faster. This optimization overlaps Host → GPU transfers of cached KV with GPU computation on the new tokens. We want the GPU to compute as data arrives instead of waiting through the transfer. HiCache therefore starts computation as soon as the first model layer’s KV is ready, while later layers’ KV continues loading.
For a three-layer model, first load the document’s layer-0 KV from Host to GPU. The GPU then processes the new question through layer 0, using that cached KV to attend to the document. At the same time, the transfer stream loads the document’s layer-1 and layer-2 KV.
Host → GPU transfers can overlap with computation because the KV can be loaded from Host one model layer at a time. Once all the KV needed by layer 0 arrives, the GPU can compute layer 0 while later layers continue loading. Each layer waits only for its own KV, rather than for every layer. An L2 hit can start this process directly. For an L3 hit, the required KV must first be read into Host as whole pages: the storage layout described above does not provide layer-by-layer reads.
Load all layers firstCompletion: 12
Overlap with computeCompletion: 10
Cache policies
Cache policies decide when to save KV and when to wait for it to load. This is a familiar tradeoff: recompute when that is faster, and load the cached data when loading is faster. Saving another copy gives later requests another chance to hit the cache, but consumes bandwidth and memory now. Waiting longer for KV may avoid recomputation, but the read itself takes time and delays the request’s start.
Prefetch
Suppose the GPU is processing request A when request B arrives. B needs a document whose KV is saved in L3. HiCache can start reading B’s KV into Host while B waits for scheduling, overlapping that read with A’s computation. When B gets its turn, it can load the KV already in Host to the GPU instead of starting the storage read then.
If B is ready to run but its L3 read is unfinished, the prefetch policy decides what happens next. Best-effort stops waiting, reuses the prefix already available, and recomputes the rest. Wait-complete keeps B waiting for the read to finish. Timeout gives the read more time, up to a limit, then proceeds with the available prefix.
Whether waiting pays off depends on two times: how long it takes to read the missing KV, and how long it takes to recompute it. For a long document, a short extra wait may save substantial prefill work. When storage reads are slow, recomputing can finish sooner. Prefetch tries to finish the read before the request runs; the policies above handle whatever remains unfinished. The diagram compares these policies as the cost of recomputation changes.
Best-effort SlowestFirst token t = 8
wait-complete FastestFirst token t = 6
timeout First token t = 7
Backup
A later request can hit Host memory only if the KV was backed up earlier. After the GPU computes a document’s KV, HiCache can copy it to Host and, with L3 enabled, retain it in storage as well. A backup adds a copy in a lower tier. The GPU copy can remain until it is evicted. The same prefix KV can therefore coexist in L1, L2, and L3, although not every prefix always has three copies. Copying every cached prefix also spends bandwidth and memory on documents that nobody asks about again.
For GPU → Host backup, HiCache provides three choices:
| Policy | Backup timing | Main tradeoff |
|---|---|---|
| Write-through | Proactively copy KV to Host when it enters the cache | Preserve more prefixes, with more backup traffic |
| Write-through-selective | Use hit counts to select hot prefixes for Host backup | Spend less bandwidth on rarely reused prefixes |
| Write-back | Back up KV to Host when it is about to be evicted from GPU memory | Defer copying, with backup work at eviction |
Hybrid-model state
Hybrid models may need sliding-window KV and recurrent state snapshots (checkpoints), alongside full-attention KV. Unified Radix Cache checks that the required states are available and determines where computation can resume. HiCache backs up these states across Host and L3, then restores them to the GPU when needed.
Backing up new recurrent snapshots
This problem concerns layers that retain history as recurrent state, such as linear attention and Mamba. Their state changes as each token arrives. A checkpoint saves that state at a particular prefix end. Full-attention layers keep per-token KV; recurrent layers keep overwriting their state, so retaining their history requires additional care.
Suppose request A is “document + question A”. Host already holds its full-attention KV and the snapshot after question A. Request B is “the same document + question B”. In this example, processing B creates a snapshot at the document’s end, before either question. A’s existing snapshot includes question A, so it cannot replace this new document snapshot.
Before: new nodes trigger backup. The insertion flow starts a backup when it adds a new cache node. During execution, a node with full-attention KV already in Host is treated as backed up and skipped. A recurrent snapshot created later for an existing prefix can therefore remain on the GPU, without a path to back up that snapshot separately to Host.
After: a new snapshot on an existing prefix also triggers backup. On the write-through path, SGLang checks the prefix’s recurrent snapshot: if it exists on the GPU but has no Host copy, it schedules a separate backup. Only the missing snapshot is copied; existing KV and snapshots stay in place. After GPU eviction, the snapshot can be restored from Host, reducing recomputation.
Snapshots at the ends of questions A and B can also be retained separately, for continuing their respective histories. The snapshot added here is at the shared document’s end: it lets a later request reuse the document and then compute its own question.
GPU · L1
Host · L2
The write-through recovery test builds new branch checkpoints, evicts them from the GPU, and requests the shared prefix again. Full recovery of the 6.4K shared prefix rises from 1/10 to 10/10 requests; the token hit rate rises from 15.74% to 64.00%. Each request has 6.4K shared tokens and 3.6K new tokens, so 64% means the entire shared prefix was recovered. See the branch-backup test.
Speculative decoding also requires backing up and restoring the draft model’s history. Otherwise, even if the target model reuses a cached prefix, missing draft history can lower the acceptance rate of proposed tokens.
Buffer-only mode
The preceding sections use Host as an L2 cache: KV stays in memory so later requests can reuse it directly. In practice, L3 has far more storage capacity than L2. If L3 is enabled but each instance has limited Host memory, Buffer-only is an optional mode. L3 retains the cached history, while Host holds only the temporary data needed for transfers.
In the diagram, instance A backs up prefix P by copying its KV from GPU to Host, then writing it to L3. Once the write completes, those Host slots are released for reuse. To restore P, instance B reads its KV from L3 into its own Host buffer, loads it to the GPU, and releases the buffer after the load completes. The GPU ↔ Host ↔ L3 transfer path stays the same; what changes is whether Host retains a copy after the transfer. As shown, buffer-only mode releases that copy.
This mode allows a smaller Host pool to handle transfers between GPU and L3. The tradeoff is that reusing a prefix missing from GPU memory requires an L3 read, without the option of hitting a retained L2 copy, and loading from L3 takes longer. With sufficient Host memory, cache mode can retain reusable KV and reduce storage reads.
Cache (default)
Buffer-only (optional)
Results
Long inputs and multi-turn conversations can outgrow GPU memory. The following benchmarks add CPU memory and external storage in turn, measuring how retaining more history changes time to first token and input throughput.
Long context · 2025
The original LooGLE evaluation tests whether lower tiers can avoid repeated prefill for long inputs. It runs DeepSeek-R1 on eight H20-3e GPUs and compares GPU-only caching, an added CPU L2, an added 3FS L3, and a separately prepopulated L3. The chart shows average TTFT and input throughput.
With ordinary L3, input throughput is 51,168 tokens/s and average TTFT is 1.83 seconds. Filling L3 with reusable KV before requests arrive lets more requests load cached prefixes immediately. Throughput reaches 76,322 tokens/s and average TTFT falls to 1.43 seconds.
Hybrid multi-turn workloads · 2026
Multi-turn conversations provide a growing working set: every round adds history that later requests may reuse. The August 2026 evaluation compares GPU L1, L1 plus CPU L2, and L1 plus L2 plus a 500 GiB Mooncake L3. DeepSeek-V4-Flash uses FULL and SWA; Inkling-Small uses FULL, SWA, and MAMBA.
L1 loses reusable history first, and L2 delays that capacity limit. Adding L3 raises DeepSeek-V4-Flash’s effective input throughput from 9.4K to 145.5K tokens/s relative to L1, with cache hits near 98%. Inkling-Small rises from 15.5K to 67.1K and finishes at a 96.8% hit rate. These results show that lower tiers can preserve reuse as hybrid-model conversation histories outgrow the smaller caches. See the multi-turn evaluation.
Input throughput counts all input tokens served per second, including cached tokens. Reusing a long prefix lets the service handle those tokens without recomputing their KV, leaving more GPU time for new input.
Further reading
This visualized tutorial was written by Yichi Zhang.