DFlash to DSpark in SGLang: 4.3× faster decode, and DeepSeek-V4 support
DFlash and DSpark are two modern speculative decoding algorithms. This blog explains in detail how these two algorithms work. Instead of the traditional AR step, both algorithms adopt a DiT architecture for the draft model. SGLang now fully supports these two algorithms: with DFlash, decode reaches 4.3× the non-speculative baseline and 1.5× MTP; with DSpark, verify windows contract from 5.24 to 2.91 tokens on mixed traffic while holding 0.88–0.97 of the acceptance ceiling, and batch-1 decode on DeepSeek-V4-Pro reaches 383.7 tok/s.
Background
To start, the basic idea behind speculative decoding: decode is memory-bound, so a target-model forward spends most of its time moving weights, and at small batch verifying several tokens costs about the same as generating one. A small drafter writes K tokens first, and the target checks them all in one forward. The accepted prefix is committed, and the verify forward emits one bonus token as a side effect. Rejected drafts only waste the drafter’s cheap compute:
The trade sours as load grows. At batch size B with K speculative tokens the target verifies B×K tokens every step. At small batch those extra tokens ride along with the weight traffic almost for free. As the batch fills the GPU, every verify token starts to cost real step time, and past a point verification costs more than it saves. The diagram below prices the same three choices at different loads:
The rest of this post is in three parts: the two algorithm generations that attack this trade, the engineering SGLang builds to serve them, and what that stack means for DeepSeek-V4.
From DFlash to DSpark
DFlash: parallel drafting with KV injection
Most speculative methods, including the EAGLE series and the native MTP heads that ship with models like DeepSeek-V4, draft autoregressively: the small model emits tokens one by one, so draft cost grows with draft length, and keeping latency low forces the drafter to stay shallow.
DFlash, developed by Z Lab, replaces the autoregressive drafter with a lightweight block diffusion model that emits the whole block in one forward. Making that work is not just swapping the drafter. A small diffusion model trained from scratch accepts poorly, and a full-size diffusion LLM is too expensive to draft with. DFlash’s answer is KV injection: the target model knows the context best, so DFlash extracts the target’s hidden states and injects them into the draft model’s KV cache at every draft layer. EAGLE-style methods feed target features only at the drafter’s input, and that signal fades as the drafter gets deeper. Injected at every layer, the context signal stays strong, and the drafter can skip modeling the context entirely and spend its capacity on predicting the block:
The two pieces contribute separately, and the paper’s ablation makes that visible. Diffusion alone already beats EAGLE-3 end to end on speed, even at lower acceptance. Injection alone raises acceptance above EAGLE-3. Together they hold acceptance at EAGLE-3’s level while drafting an order faster: on GSM8K, 4.2 accepted tokens and 3.3× end-to-end against EAGLE-3’s 4.2 and 2.1×. On Qwen3.5-397B-A17B, the jointly released drafter reaches 4.3× the non-speculative baseline and 1.5× MTP at concurrency 1 on HumanEval:
DSpark: a sequential head and a verify budget
A purely parallel block leaves two things on the table. First, the block’s positions are drafted without intra-block dependencies, so quality decays toward the tail of the block; the DSpark paper calls this suffix decay. Second, the verify side is untouched: every request still pays for a full-block verify, which is exactly the B×K cost from the Background at high concurrency.
DSpark, from DeepSeek, addresses both ends with three draft-side pieces. The block drafter still emits a gamma-token block in one forward, but a lightweight sequential head (a Markov or RNN head) conditions each position on the previous token; this is what semi-autoregressive means, and it is the piece that counters suffix decay.
To tackle the full-block verify problem at high concurrency, DSpark adopts the confidence head. The head is a linear projection plus a sigmoid on the drafter’s hidden state, and it scores each drafted token’s chance of surviving verification. Its training label is the measured gap between the drafter’s output distribution and the target’s at that position, so the score simply tracks how predictable the next tokens are: formulas and code pin the next token down and score high, while free-form prose scores low. The product across the block is the block’s survival probability, and Sequential Temperature Scaling (STS) calibrates it to match true acceptance rates.
Those survival scores are what makes the verify length schedulable per request: walk down the block’s survival curve and stop where the expected accepted tokens no longer cover the marginal cost of verifying one more token. The diagram below follows one request through a draft step:
Across Qwen3-4B, 8B, and 14B targets, the paper reports 16.3%, 18.4%, and 18.3% higher macro-average accepted length than DFlash. The rest is a serving problem: someone has to price the marginal cost, cut the budgets every step, and make a batch of unequal verify lengths run fast on real GPUs. That is the SGLang part.
How SGLang serves DSpark
SGLang integrated DSpark in #30261 (merged July 2026), building the drafter directly on the DFlash draft model, for both dense and sparse targets such as Qwen3 and DeepSeek-V4. Across a full concurrency sweep on DeepSeek-V4-Flash, the integration holds the best throughput/latency frontier, ahead of both MTP and the non-speculative floor:
The rest of this section walks through the engineering that turns the paper’s schedule into wall-clock time.
The confidence scheduler and verify modes
In one sentence: this optimization turns “how many tokens to verify” from a global constant into a per-request, per-step decision.
Every step, SGLang’s confidence scheduler runs the budget rule from the previous section for each request: walk down the survival curve and stop where the expected gain no longer covers the marginal cost. SGLang measures the marginal cost on this server: how much each extra verify token adds to the step time, stored as a cost table. The window length falls out of that comparison: a request whose drafts score high keeps a long window, a request whose scores drop fast gets trimmed to two or three tokens, and higher load raises the marginal cost, tightening every window together:
Verification runs in three modes, and the split trades speed against visibility. compact verifies only the scheduled window; it is the production path, and it is where the budget’s savings land. Its blind spot: the trimmed positions are never verified, so you cannot see how many tokens a full verify would have accepted, or judge whether the scheduler trims well. static verifies the full block; it is the no-trim baseline for A/B comparisons. cap-accept exists for measurement: it pays static’s verify cost but commits only the window, so its output matches compact exactly while it records what the trim gave up. In practice, traffic runs on compact, and cap-accept serves as an audit arm for checking trim quality; the 0.88–0.97 utilization numbers in the mixed-traffic section below come from exactly such runs:
Ragged verify under full CUDA graphs
CUDA graphs demand fixed input shapes, but after trimming every request verifies a different length, and padding is not an option here: it would put the trimmed block back. SGLang front-packs the scheduled verify tokens into one [total_tokens, hidden] buffer, the same layout a transformer already uses for its per-row matmuls. A cu_seqlens-style varlen input marks where each request’s segment starts, so every token attends to its own request’s KV and the backend’s existing kernels handle it (flash_mla on DeepSeek-V4). The graph is then keyed on the packed total, rounded up to the nearest captured tier. When budgets tighten, the total drops to a smaller tier, and the replayed graph has genuinely fewer attention and MLP rows. Under data-parallel attention the ranks share the largest tier and step down together:
Scheduling results
How much does the dynamic budget actually buy? The blog compares compact against no-trim, which runs the static full-block schedule through the same ragged path, on two example workloads that differ in acceptance. The win is primarily a high-batch effect. At batch size 1 the target verify barely slows down with more tokens, so trimming saves little and the two arms tie. As concurrency grows and throughput starts to plateau, trimming shortens the step and compact pulls ahead. The gap is larger, and opens earlier, on the lower-accept workload, because lower acceptance leaves more tail to trim.
This scheduler is a first, vanilla version, so the trend is the result, not the exact numbers. Homogeneous sweeps also hide its real point: two requests in the same batch should not get the same window if one is far more predictable than the other. On a mix of gsm8k, arena-hard, and poetry, the window contracts with difficulty, 5.24 to 3.78 to 2.91 tokens, while utilization against the ceiling stays at 0.88–0.97. About 55% of gsm8k steps fill the full window of six; about 80% of poetry steps use three or fewer:
Zero-overhead scheduling and kernels
Turning the schedule into wall-clock time takes two kinds of engineering. On the kernel side, SGLang rewrote the clusters of tiny ops around the verify as fused Triton kernels: the compact scatter, the SWA page-index, the verify-length top-k schedule, and the ragged-window packing. The block drafter’s sampling path folds into fused kernels, and its matrix multiplication is sharded. In one example profile, the work outside the target verify shrinks by 1.7 ms, against a 7.3 ms verify.
On the scheduling side, DSpark drops straight into SGLang’s zero-overhead (overlap) scheduler with almost no special-casing. The spec-v2 runtime was introduced together with the DFlash integration, where its overlap scheduling added another 33% throughput on top of the initial implementation. It overlaps the next step’s scheduling with the current forward on separate streams, and DSpark joins as a first-class worker: forward outputs come back as async futures, cross-iteration ordering rides the runtime’s device-side barrier, and on-device page tables mean no per-step host sync. The confidence relay uses the same channel, read two steps back. The decode loop then runs with no per-step bubble, about 1.5× tighter than with the overlap scheduler off:
Serving DeepSeek-V4 with SGLang
DeepSeek-V4 ships as the 1.6T Pro and the 284B Flash, and SGLang has served
the family since day 0, with prefix caching, kernels, and parallelism built for
its hybrid attention path. Speculation sits on top as a serving-time choice:
--speculative-algorithm picks EAGLE, which runs the MTP head inside the
checkpoint, or DFLASH, or DSPARK, and none of them changes the target model’s
architecture or output distribution.
| Speculation | Target · hardware | Accept length | Single-user decode |
|---|---|---|---|
| EAGLE (MTP head) | V4-Pro · B200 TP8 | ~2.5 | 199 tok/s |
| EAGLE (MTP head) | V4-Flash · H200 TP4 | ~2.5 | 266 tok/s |
| DSpark | V4-Pro · B300 TP8 | ~5 | 383.7 tok/s |
The first two rows come from the day-0 blog, the third from the DSpark blog. The hardware generations differ, so read this as a reference table, not a controlled A/B; the same-server comparison is the frontier figure earlier in this post.
Acknowledgments
The original blogs thank the DSpark authors and DeepSeek for the algorithm and the models, and Z Lab, Modal, and the SGLang team for DFlash and Spec V2.
Further reading
- LMSYS Blog: DSpark in SGLang: Speculative Decoding with Confidence-Driven, Variable-Length Verification
- LMSYS Blog: The next generation of speculative decoding: DFlash and Spec V2
- LMSYS Blog: DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles
- sglang#30261: [Spec] Add DSpark: confidence-scheduled speculative decoding
- sglang#30344: [Tracking] DSpark in SGLang: roadmap
This visualized tutorial was written by Yichi Zhang.