DeepSeek AI Released DeepSeek-V4.1-Flash with 1M Context, FP4 KV Cache, and Cross-Layer Attention Reuse
Long-horizon brokers have turned LLM serving into an input-heavy workload. Repeated prefills and million-token contexts depart KV caches that pressure HBM, SSD capability, and bandwidth. DeepSeek AI constructed its latest launch round that actual bottleneck. DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts mannequin with 552B spine parameters, 196B further Engram parameters, and a 1M-token context window. It prompts 8B parameters per token throughout prefill and 16B throughout decode. The primary quantity is a world KV cache footprint of 890 bytes per token, about 1/4 of DeepSeek-V4-Flash and roughly 437x smaller than DeepSeek-V1.
Is it deployable? Yes. Open weights ship underneath an MIT license with vLLM, SGLang, and Transformers paths on Hugging Face, and the analysis crew describes a public API with low, excessive, and max reasoning tiers.
Causal Encoder-Decoder: Half the Prefill
The 40-layer spine is cut up right into a 20-layer causal encoder and a 20-layer decoder. Inspired by YOCO, the decoder doesn’t compute its personal world KV. Instead, per-layer projection weights derive it from the ultimate encoder hidden state. Prompt tokens subsequently cease on the encoder, which almost halves prefill compute. Sliding-window consideration (SWA) with a 128-token window nonetheless runs in each layer, so decoder SWA states are rebuilt by replaying solely the final 128 immediate tokens. The analysis crew calls this Decoder SWA Bounded Replay.
Compressed Sparse Attention 2 (CSA2)
DeepSeek-V4 combined CSA with Heavily Compressed Attention. V4.1-Flash makes use of pure CSA2 and assaults cache dimension alongside the layer axis. Each CSA2 layer is statically assigned considered one of 3 modes:
- Full: computes its personal primary KV, tasks indexer Ok from it, and selects recent Top-512 indices.
- Reindex: reuses primary KV and indexer Ok from the final Full layer however rescores them with its personal indexer Q.
- Reuse: reuses each the principle KV and the newest Top-Ok indices, skipping the indexer fully.
Every layer retains its personal primary Q and SWA KV. The 18 CSA2 encoder layers use a compression ratio of two in 3 teams of 6 (1 Full, 5 Reuse). The 20 decoder layers use ratio 1 in 5 teams of 4: the primary is Full plus 3 Reuse, the remaining Reindex plus 3 Reuse. A Hierarchical Sparse Indexer within the decoder lets the Full layer construct a candidate pool of as much as 16,384 positions (2,048 blocks of 8), so later Reindex layers rating a bounded set as a substitute of all the context.
FP4 KV, Bounded Replay, and Other Extensions
The primary KV cache is quantized to E2M1 with one E4M3 scale per 16 channels, following NVFP4 with out its world scale. This is launched by way of quantization-aware coaching in post-training and almost halves storage towards V4’s FP8 cache.
At the deployment degree, SWA KV is not persevered to SSD. It lives in a distributed pool carved from 10% of host DRAM with a TTL of minutes, whereas world KV retains a assured 72-hour lifetime. On a miss, Encoder SWA Bounded Replay recomputes solely 128 tokens as a substitute of layers occasions window.
Other adjustments embody Single-Pass mHC, which shifts input-mixing coefficients by one block so a fused Mega-mHC kernel can halve activation reminiscence visitors, the Engram conditional reminiscence module at layers 1 and 14, DSpark speculative decoding skilled after pre-training with the spine frozen, and head-wise Muon. Single-token decode FLOPs rise by only one/4 when context grows from 4K to 1M.
Training and Results
Pre-training covers 45T multimodal tokens at a 7:1 text-to-multimodal ratio. Sparse consideration is skilled from scratch at 64K sequence size with no dense warmup, and context is prolonged to 1M at 34T tokens. The base mannequin matches DeepSeek-V4-Pro-Base on world data and coding whereas utilizing 1/3 of the entire and 1/4 of the activated parameters.
Post-training introduces no new algorithms. Gains come from large-scale synthesis of verifiable agent duties, RL throughout heterogeneous scaffolds (Claude Code, Codex, OpenCode, Pi, mini-SWE, DeepSeek Harness), and on-policy distillation from over 40 academics. Selected max-effort outcomes:
| Benchmark | DS-V4.1-Flash | DS-V4-Flash | Opus-5 | GPT-5.6 Sol |
|---|---|---|---|---|
| Terminal-Bench 2.1 | 90.6 | 82.7 | 89.1 | 88.8 |
| DeepSWE v1.1 | 74.2 | 54.4 | 74.0 | 73.0 |
| Terminal-Bench 4.0 | 31.2 | 7.0 | 51.8 | 39.9 |
| Automation-Bench | 54.8 | 37.7 | 50.3 | 45.8 |
| GPQA Diamond | 90.9 | 89.9 | 93.4 | 94.1 |
| Codeforces (score) | 3471 | 3289 | n/a | n/a |
Interactive Explainer
Key Takeaways
- Global KV cache falls to 890 bytes per token, about 1/4 of V4-Flash and 437x beneath V1.
- CED runs solely 20 encoder layers in prefill, activating 8B parameters towards 16B in decode.
- CSA2 shares primary KV, indexer Ok, and Top-Ok indices throughout layers in Full, Reindex, and Reuse modes.
- FP4 primary KV plus SWA Bounded Replay reduce persistent cache to about 1/8 of V4-Flash.
- Beats Opus-5 and GPT-5.6 Sol on Terminal-Bench 2.1 and DeepSWE v1.1 with MIT weights.
Check out the Model on Hugging Face and the Technical Report. Also, be happy to comply with us on Twitter and don’t overlook to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to accomplice with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so on.? Connect with us
The put up DeepSeek AI Released DeepSeek-V4.1-Flash with 1M Context, FP4 KV Cache, and Cross-Layer Attention Reuse appeared first on MarkTechPost.
