DFlash Speculative Decoding Drafts Whole Token Blocks in Parallel for Up to 15x Higher Throughput on NVIDIA Blackwell

Autoregressive giant language fashions generate textual content one token at a time. Each token waits for the one earlier than it. This serial loop leaves trendy GPUs underused and retains inference gradual. The value grows worse with lengthy Chain-of-Thought reasoning fashions. Their prolonged outputs make latency the dominant a part of era.

Speculative decoding is the usual repair. A small draft mannequin proposes future tokens. The giant goal mannequin verifies these tokens in parallel. Accepted tokens are saved, so the output stays lossless. But most strategies, together with the state-of-the-art EAGLE-3, nonetheless draft autoregressively. That serial drafting caps real-world speedups close to 2–3×.

DFlash, launched by analysis workforce from UC San Diego workforce (z-lab), takes a special route. It is a light-weight block diffusion mannequin constructed for drafting. Instead of drafting tokens one by one, it proposes a complete block in a single ahead move. The goal mannequin then verifies that block in parallel.

The analysis workforce stories over 6× lossless acceleration throughout a variety of fashions and duties. It reaches up to 2.5× greater speedup than EAGLE-3. On NVIDIA Blackwell, NVIDIA engineering workforce stories up to 15× greater throughput for gpt-oss-120b. That determine holds on the identical person interactivity goal.

What block diffusion drafting modifications

Block diffusion fashions denoise a block of masked tokens without delay. They mix parallel era with autoregressive block construction. DFlash applies this concept solely to the drafting stage. Verification stays with the trusted autoregressive goal mannequin.

This break up issues for high quality. Standalone diffusion LLMs typically path autoregressive fashions on accuracy. They additionally want many denoising steps, which slows their uncooked inference velocity. DFlash sidesteps each issues. The draft solely wants to be ok to be accepted. The goal’s parallel verification ensures the ultimate output distribution.

A second profit is drafting value. An autoregressive drafter’s value grows linearly with the variety of speculative tokens. A diffusion drafter generates all tokens in one parallel move. So drafting latency stays largely flat because the block grows. This frees DFlash to use deeper, extra expressive draft fashions with out including latency.

This separates DFlash from earlier diffusion-drafter work. Methods like DiffuSpec and SpecDiff-2 used huge 7B drafters, capping speedups close to 3–4×. DFlash as a substitute makes use of a small five-layer drafter (eight layers for Qwen3-Coder).

The “goal is aware of greatest” perception

DFlash’s core thought is straightforward: the goal is aware of greatest. Large autoregressive fashions’ hidden options encode details about a number of future tokens. DFlash extracts hidden states from a number of goal layers. It fuses them into one compact goal context characteristic. This characteristic then circumstances the draft mannequin.

DFlash injects this characteristic in a different way than EAGLE-3. EAGLE-3 fuses goal options into the draft’s enter embeddings solely. As draft depth grows, that sign will get diluted. DFlash as a substitute injects the characteristic into the Key and Value projections of each draft layer. The projected options sit in the draft’s KV cache and persist throughout drafting iterations.

This KV injection lets acceptance size scale with draft depth. A five-layer DFlash drafter producing 16 tokens beats EAGLE-3 producing 8 tokens. It is each lower-latency and higher-acceptance in the paper’s exams. The draft mannequin successfully turns into a diffusion adapter on high of the goal.

Two speedup numbers, measured in a different way

The DFlash analysis’s 6× is single-stream lossless acceleration. On Qwen3-8B with grasping decoding (Transformers backend), DFlash averages a 4.86× speedup. EAGLE-3 averages 1.76× at tree dimension 16 and a pair of.02× at tree dimension 60. DFlash peaks at 6.08× on MATH-500 (τ = 7.87) and averages τ = 6.49 throughout duties.

NVIDIA’s 15× is throughput at a set interactivity goal. It applies to gpt-oss-120b on eight NVIDIA Blackwell GPUs in a DGX B300 system, utilizing TensorRT-LLM. At the five hundred–600 tokens/sec per-user vary, DFlash serves greater than 15× the throughput of autoregressive decoding. That is about 1.5× greater than EAGLE-3 on the identical level.

The desk beneath reveals the paper’s per-task speedups on Qwen3-8B at temperature 0 (Transformers backend).

A separate NVIDIA Speed-Bench comparability measures interactivity speedups at matched concurrency. On gpt-oss-120b, DFlash averages 2.3× versus EAGLE-3’s 1.7×. On Llama 3.1 8B Instruct, DFlash averages 2.8× versus EAGLE-3’s 2.2×.

Use circumstances with examples

DFlash targets latency-sensitive serving the place token-by-token era hurts. Three patterns match properly:

• Coding brokers: Code era wants quick, interactive responses. On Gemma 4 31B with vLLM, NVIDIA stories up to 5.8× on Math500 at concurrency 1. HumanEval reaches 5.6×. Faster drafts imply shorter wait occasions inside agent loops.
• Reasoning fashions: Long Chain-of-Thought traces dominate era time. With pondering mode enabled, DFlash holds roughly 4.5× underneath grasping decoding on Qwen3-4B and Qwen3-8B. Under sampling, it holds about 3.9×. This cuts the price of lengthy reasoning outputs.
• Serving and throughput: DFlash additionally raises serving throughput. On SGLang with a B200 GPU, it reaches up to 5.1× on Qwen3-8B (Math500, concurrency 1). Gains taper as concurrency rises however keep constructive, so serving value nonetheless drops.

Running DFlash

DFlash ships with checkpoints and framework assist, so adoption wants little code. On vLLM, you swap an EAGLE-3 config for a DFlash one. No utility refactoring is required.

vllm serve Qwen/Qwen3.5-27B 
  --speculative-config '{"methodology": "dflash", "mannequin": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 15}' 
  --attention-backend flash_attn 
  --max-num-batched-tokens 32768

The Transformers backend helps Qwen3 and LLaMA-3.1 fashions. It exposes a spec_generate name that pairs a draft mannequin with a goal mannequin.

from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer

draft = AutoModel.from_pretrained(
    "z-lab/Qwen3-8B-DFlash-b16", trust_remote_code=True,
    dtype="auto", device_map="cuda:0").eval()
goal = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B", dtype="auto", device_map="cuda:0").eval()
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

messages = [{"role": "user", "content": "How many positive whole-number divisors does 196 have?"}]
input_ids = tokenizer.apply_chat_template(
    messages, return_tensors="pt", add_generation_prompt=True,
    enable_thinking=False).to(draft.gadget)

output = draft.spec_generate(
    input_ids=input_ids, max_new_tokens=2048, temperature=0.0,
    goal=goal, stop_token_ids=[tokenizer.eos_token_id])
print(tokenizer.decode(output[0], skip_special_tokens=False))

Key Takeaways

• DFlash drafts a whole token block in one ahead move, not one token at a time.
• It injects goal hidden options into each draft layer’s KV cache, scaling acceptance size with depth.
• Research Paper’s metrics: up to 6.08× lossless speedup on Qwen3-8B; NVIDIA check: up to 15× throughput on Blackwell at mounted interactivity.
• A light-weight five-layer drafter replaces the 7B drafters that capped earlier diffusion strategies close to 3–4×.

Interactive Explainer

Check out the Project page, Paper (arXiv 2602.06036), GitHub, Hugging Face checkpoints and NVIDIA blog. Also, be at liberty to comply with us on Twitter and don’t overlook to be part of our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The submit DFlash Speculative Decoding Drafts Whole Token Blocks in Parallel for Up to 15x Higher Throughput on NVIDIA Blackwell appeared first on MarkTechPost.