NVIDIA Releases Nemotron-Labs-3-Puzzle-75B-A9B: A Compressed Hybrid MoE LLM Delivering 2.03x Server Throughput at Matched User Throughput
Large hybrid MoE fashions like Nemotron-3-Super are correct however costly to serve. Their energetic parameters, KV cache, and Mamba state cap what number of customers a node can maintain at a given per-user token fee. NVIDIA AI group has launched Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super. The guardian mannequin has 120.7B complete and 12.8B energetic parameters. The compressed mannequin has 75.3B complete and 9.3B energetic parameters.
The deployment goal was fastened earlier than the structure search started. Target one was 2x server throughput at 100 tokens per second per consumer. Target two was 8 concurrent 1M-token requests on a single H100. Three checkpoints on Hugging Face: BF16, FP8, and NVFP4.
TL;DR
- 120.7B/12.8B energetic compresses to 75.3B/9.3B energetic, with the 88-block hybrid structure preserved.
- 8xB200 complete throughput rises 1.60x to 2.14x over Super at matched NVFP4 and matched consumer throughput.
- Single-H100 1M-token concurrency goes 1 to eight, pushed by a 70 GB to 44.5 GB weight drop.
- Iterative Puzzle beats single-step Puzzle by 0.57 common factors at the identical compression goal.
- Arena-Hard-V2 (-4.2) and SWE-Bench (-2.6) are the actual prices; RULER and AA-LCR barely transfer.
Nemotron-Labs-3-Puzzle-75B-A9B
Nemotron-3-Super is a hybrid Mamba-Transformer MoE mannequin. Puzzle-75B-A9B preserves the guardian’s block structure precisely. It has 88 blocks: 40 Mamba, 40 MoE, and eight consideration blocks.
What modified is capability inside these blocks:
| Quantity | Super | Puzzle-75B-A9B | Ratio |
|---|---|---|---|
| Total parameters | 120.7B | 75.3B | 62.4% |
| Active parameters | 12.8B | 9.3B | 73.1% |
| Mamba SSM state dimension | 128 | 96 | 75% |
| MoE routed knowledgeable intermediate dimension | 2688 | 1280-2688 | Mean 59.9% |
| Activated routed specialists per token | 22 | 4-18 | Mean 50% |
| Active routed knowledgeable capability (relative) | 100% | 8.7%-62.3% | Mean 30.9% |
The variety of routed specialists, the shared knowledgeable dimension, and the MoE latent dimension are unchanged. Attention layers have been left untouched. The proposed analysis’s acknowledged motive is that Nemotron-3-Super is already very KV-cache environment friendly. Mamba layers have been pruned uniformly, as a result of inference frameworks don’t assist a distinct SSM state dimension per layer.
The consequence just isn’t a uniformly scaled-down instructor. The above determine reveals the allocation throughout depth. Puzzle preserved capability in chosen center and late layers, and lower onerous elsewhere.
Benchmark and Performance
The beneath desk reviews Pareto-optimal complete throughput on a single 8xB200 node, with single-step decoding.
| Scenario (in/out) | UT flooring | Super (tok/s) | Puzzle-75B-A9B (tok/s) | Boost |
|---|---|---|---|---|
| 50K / 2K | >= 100 | 5,128 | 8,210 | 1.60x |
| 50K / 2K | >= 125 | 3,784 | 6,412 | 1.69x |
| 50K / 2K | >= 150 | 2,532 | 4,523 | 1.79x |
| 8K / 64K | >= 100 | 20,939 | 42,601 | 2.03x |
| 8K / 64K | >= 125 | 13,074 | 27,918 | 2.14x |
| 8K / 64K | >= 150 | 8,522 | 18,047 | 2.12x |
Both fashions have been served at matched NVFP4 weights, FP8 KV cache, and FP16 Mamba state. The hole due to this fact displays compression, not a change in numeric format. The prefill-heavy 50K/2K regime positive aspects least. The decode-heavy 8K/64K regime positive aspects most.
On a single 8xH100 node at UT = 100, the positive aspects are smaller. They are 1.91x on 50K/2K and 1.82x on 8K/64K. Both fashions there use FP8 weights, FP8 KV cache, and FP32 Mamba state.
On a single H100 at 1M context, the binding constraint flips from compute to reminiscence. Super’s NVFP4 weights occupy about 70 GB of the 80 GB HBM finances. Each 1M-token request provides about 4 GB of KV cache. Effective concurrency is due to this fact 1.
Puzzle-75B-A9B’s NVFP4 weights occupy about 44.5 GB. Attention structure is unchanged, so per-request KV price is unchanged. Concurrency at 1M rises to eight. Aggregate decode throughput at that concurrency is roughly 4x Super’s single-request throughput. Prefill of a 990K-token immediate is about 1.2x sooner.
How Iterative Puzzle Works
Puzzle is a decomposed neural structure search framework, applied right here as Puzzletron. It defines a discrete search area of different layer implementations. Each various will get a top quality rating. A mixed-integer program then selects one various per layer underneath a deployment constraint.
Three pruning strategies kind the search area:
- Intermediate channel pruning: Channels inside every routed knowledgeable are ranked by contribution to the knowledgeable’s output. All specialists inside one MoE layer are pruned to a uniform dimension, for kernel compatibility.
- Top-k discount: The variety of specialists a token is routed to varies per layer, as much as the guardian’s ok=22.
- Mamba SSM pruning: The SSM state dimension drops from 128 to 96 channels.
The SSM result’s measured. Dropping 128 channels to 96 speeds the SSM kernel 1.2x to 1.3x throughout decode. This holds at batch sizes between 8 and 512. Channels have been ranked by estimated contribution to the Mamba layer output. The estimate averaged over 67M tokens of validation information. Appendix A reveals this beats random channel choice underneath aggressive pruning.
The unique formulation assumes substitute high quality impacts are roughly additive. Each candidate block is scored contained in the unmodified guardian. That ignores higher-order interactions between replacements.
Iterative Puzzle alternates bounded compression with quick data distillation restoration. It builds a sequence M0, M1, … MR as a substitute of leaping to the goal. Scores are recomputed towards the present compressed mannequin, not the unique guardian.
Three levels have been used:
- MoE weights to 75% of instructor capability, Mamba SSM state to 75%. Healed for 24B tokens.
- MoE weights to 60% of instructor capability. Healed for 43.2B tokens.
- Activated routed-expert finances to 50%, allotted heterogeneously. Healed for 52.8B tokens.
The above desk compares this towards a single-step Puzzle baseline at the identical goal. The three-step process averages 69.05 throughout ten benchmarks, towards 68.48. Gains seem on MMLU-Pro, GPQA, HLE, AA-LCR, LiveCodeBench, SciCode, and RULER-256K. IFBench-Instruction fell 0.2 factors and IFBench-Prompt fell 0.5.
Recovery: Distillation, RL, and Verbosity
Knowledge distillation ran on 30% pretraining information and 70% SFT information from Nemotron-3-Nano. During the Puzzle section, KD used a 32K sequence size. Recovery then educated at 128K, and scaled to 512K. The finances was as much as 100B tokens, with a 16M-token world batch, in Megatron-LM.
RL post-training adopted Stage 2 of the Nemotron-3-Super RL pipeline, centered on software program engineering. Phase 2.1 did single-step tool-use comparability. Phase 2.2 moved to end-to-end sandbox RL, the place brokers run as much as 200 turns. Both phases used a KL penalty of 0. The group swept studying charges, then averaged the ensuing weights.
The above Figure 4 reveals what every stage contributed. Short-context KD recovers most classes to over 97% of Nemotron-3-Super. Long-context KD then lifts long-input and long-generation benchmarks particularly. The analysis group states that RL’s impression in these experiments was small.
Verbosity is the quiet element. After the final Puzzle iteration, the mannequin generated 132% of Super’s token rely. That fell to 99% after the complete restoration pipeline.
Deployment: Quantization and Multi-Token Prediction
Two post-training quantization recipes have been produced: FP8 W8A8 targets Hopper and NVFP4 W4A4 targets Blackwell.
| Component | BF16 baseline | FP8 checkpoint | NVFP4 checkpoint |
|---|---|---|---|
| Sparse and shared MoE GEMMs | BF16 | FP8 | NVFP4 |
| Mamba GEMMs | BF16 | FP8 | FP8 |
| Mamba SSM cache | FP32 | FP32 | FP16+SR |
| KV cache | FP8 | FP8 | FP8 |
| Router | FP32 | FP32 | FP32 |
| Attention QKV/output, MoE latent projections, LM head | BF16 | BF16 | BF16 |
Both recipes calibrated on 256 post-training SFT samples. NVFP4 used max calibration, not the AutoQuantize sensitivity search used for Super. The ensuing checkpoint is barely extra aggressively quantized, and carried out equally.
NVFP4 just isn’t natively supported on Hopper. It continues to be used for the 1M-context H100 goal, as a result of HBM capability binds there.
Puzzle-75B-A9B inherits a shared MTP head from Super. Parameters are shared throughout MTP steps, so one head applies recursively at inference. Transferring Super’s educated head immediately gave comparable acceptance lengths.
The analysis group then identifies a training-inference mismatch. Teacher-forced MTP coaching feeds the complete shifted hidden-state sequence. Autoregressive drafting as a substitute feeds a combination of target-model and MTP-generated hidden states. Acceptance charges fall at deeper draft positions.
Continued coaching on the transferred head addresses this. On SPEED-Bench at draft size 7, common acceptance size rose from 3.45 to 4.34. That is roughly 25% to 30%, concentrated at later draft positions. Unlike Super, the NVFP4 checkpoint barely degrades: 4.31 towards 4.34.
Where Compression Helps and Where It Hurts
| Benchmark (BF16) | Super | Puzzle-75B-A9B | Delta |
|---|---|---|---|
| MMLU-Pro | 83.8 | 82.4 | -1.4 |
| AIME25 (no instruments) | 92.2 | 89.7 | -2.5 |
| GPQA (no instruments) | 80.5 | 78.6 | -1.9 |
| LiveCodeBench | 82.1 | 81.1 | -1.0 |
| SciCode (subtask) | 42.3 | 40.6 | -1.7 |
| SWE-Bench (OpenHands) | 59.5 | 56.9 | -2.6 |
| Arena-Hard-V2 | 72.8 | 68.6 | -4.2 |
| AA-LCR | 56.8 | 56.9 | +0.1 |
| RULER 1M | 93.9 | 92.2 | -1.7 |
| MMLU-ProX | 79.5 | 77.5 | -2.0 |
The analysis paper’s personal abstract is that instruction-following and agentic evaluations lose most. Arena-Hard-V2 is the worst case, at -4.2 factors. RULER stays inside roughly 1 to 2 factors at 256K, 512K, and 1M.
Three BF16 outcomes don’t regress. AA-LCR positive aspects 0.1, Scale AI Multi-Challenge ties at 56.6, and TauBench Telecom positive aspects 0.4.
NVFP4 prices little on high of compression. On RULER 1M the NVFP4 checkpoint scores 93.2, above BF16’s 92.2. HLE is the clearest NVFP4 price, dropping from 16.5 to fifteen.7. FP8 outcomes sit in Appendix E, and monitor BF16 carefully. SWE-Bench just isn’t reported for the FP8 checkpoint.
Use Cases
- Ultra-long-context RAG on one GPU: A doc evaluation service at 1M context goes from 1 concurrent request to eight. Aggregate decode throughput at that concurrency is roughly 4x.
- Interactive coding assistants: At UT >= 100 tok/s within the 8K/64K regime, one node serves 2.03x the tokens. Adjusted for verbosity, that’s 2.16x the finished requests per minute.
- Prefill-heavy doc pipelines: The 50K/2K regime positive aspects only one.60x. Compression helps much less when immediate processing dominates compute.
- Agentic SWE loops: Check the two.6-point SWE-Bench hole towards your process combine. RL restoration focused this functionality, and solely partly restored it.
Deployment Explorer
Strengths and Weaknesses
Strengths
- 1.60x to 2.14x complete throughput over Super at matched NVFP4 and matched consumer throughput
- 1M-token concurrency on a single H100 rises from 1 request to eight
- MTP acceptance size improves from 3.45 to 4.34 on SPEED-Bench at draft size 7
- Long-context accuracy holds inside 1 to 2 factors on RULER at 256K, 512K, and 1M
- Generation verbosity ends at 99% of Super, so token positive aspects survive at request degree
- Three checkpoints revealed: BF16, FP8, and NVFP4
Weaknesses
- Arena-Hard-V2 drops 4.2 factors and SWE-Bench drops 2.6 factors
- RL restoration had a small measured impression, which the paper states immediately
- Mamba pruning is uniform, as a result of frameworks can not fluctuate SSM state dimension per layer
- Latent-dimension pruning was dropped: NVFP4 MoE kernels want a latent dim that may be a a number of of 512
- Prose and tables disagree on a number of throughput multiples on this v2 preprint
- Disaggregated prefill positive aspects are solely 5% to 7%, and add serving complexity
Check out the Paper here. Also, be happy to observe us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us
The put up NVIDIA Releases Nemotron-Labs-3-Puzzle-75B-A9B: A Compressed Hybrid MoE LLM Delivering 2.03x Server Throughput at Matched User Throughput appeared first on MarkTechPost.
