Meet Nemotron Labs 3 Puzzle 75B A9B: A Compressed Hybrid MoE LLM Delivering 2.03x Server Throughput
Large hybrid MoE fashions like Nemotron-3-Super are correct however costly to serve. Their lively parameters, KV cache, and Mamba state cap what number of customers a node can maintain at a given per-user token price. NVIDIA AI staff has launched Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super. The guardian mannequin has 120.7B complete and 12.8B lively parameters. The compressed mannequin has 75.3B complete and 9.3B lively parameters.
The deployment goal was mounted earlier than the structure search started. Target one was 2x server throughput at 100 tokens per second per person. Target two was 8 concurrent 1M-token requests on a single H100. Three checkpoints on Hugging Face: BF16, FP8, and NVFP4.
TL;DR
- 120.7B/12.8B lively compresses to 75.3B/9.3B lively, with the 88-block hybrid format preserved.
- 8xB200 complete throughput rises 1.60x to 2.14x over Super at matched NVFP4 and matched person throughput.
- Single-H100 1M-token concurrency goes 1 to eight, pushed by a 70 GB to 44.5 GB weight drop.
- Iterative Puzzle beats single-step Puzzle by 0.57 common factors on the identical compression goal.
- Arena-Hard-V2 (-4.2) and SWE-Bench (-2.6) are the true prices; RULER and AA-LCR barely transfer.
Nemotron-Labs-3-Puzzle-75B-A9B
Nemotron-3-Super is a hybrid Mamba-Transformer MoE mannequin. Puzzle-75B-A9B preserves the guardian’s block format precisely. It has 88 blocks: 40 Mamba, 40 MoE, and eight consideration blocks.
What modified is capability inside these blocks:
| Quantity | Super | Puzzle-75B-A9B | Ratio |
|---|---|---|---|
| Total parameters | 120.7B | 75.3B | 62.4% |
| Active parameters | 12.8B | 9.3B | 73.1% |
| Mamba SSM state measurement | 128 | 96 | 75% |
| MoE routed professional intermediate measurement | 2688 | 1280-2688 | Mean 59.9% |
| Activated routed consultants per token | 22 | 4-18 | Mean 50% |
| Active routed professional capability (relative) | 100% | 8.7%-62.3% | Mean 30.9% |
The variety of routed consultants, the shared professional measurement, and the MoE latent measurement are unchanged. Attention layers have been left untouched. The proposed analysis’s said motive is that Nemotron-3-Super is already very KV-cache environment friendly. Mamba layers have been pruned uniformly, as a result of inference frameworks don’t help a special SSM state measurement per layer.
The outcome will not be a uniformly scaled-down instructor. The above determine reveals the allocation throughout depth. Puzzle preserved capability in chosen center and late layers, and reduce arduous elsewhere.
Benchmark and Performance
The beneath desk studies Pareto-optimal complete throughput on a single 8xB200 node, with single-step decoding.
| Scenario (in/out) | UT flooring | Super (tok/s) | Puzzle-75B-A9B (tok/s) | Boost |
|---|---|---|---|---|
| 50K / 2K | >= 100 | 5,128 | 8,210 | 1.60x |
| 50K / 2K | >= 125 | 3,784 | 6,412 | 1.69x |
| 50K / 2K | >= 150 | 2,532 | 4,523 | 1.79x |
| 8K / 64K | >= 100 | 20,939 | 42,601 | 2.03x |
| 8K / 64K | >= 125 | 13,074 | 27,918 | 2.14x |
| 8K / 64K | >= 150 | 8,522 | 18,047 | 2.12x |
Both fashions have been served at matched NVFP4 weights, FP8 KV cache, and FP16 Mamba state. The hole subsequently displays compression, not a change in numeric format. The prefill-heavy 50K/2K regime positive factors least. The decode-heavy 8K/64K regime positive factors most.
On a single 8xH100 node at UT = 100, the positive factors are smaller. They are 1.91x on 50K/2K and 1.82x on 8K/64K. Both fashions there use FP8 weights, FP8 KV cache, and FP32 Mamba state.
On a single H100 at 1M context, the binding constraint flips from compute to reminiscence. Super’s NVFP4 weights occupy about 70 GB of the 80 GB HBM price range. Each 1M-token request provides about 4 GB of KV cache. Effective concurrency is subsequently 1.
Puzzle-75B-A9B’s NVFP4 weights occupy about 44.5 GB. Attention format is unchanged, so per-request KV price is unchanged. Concurrency at 1M rises to eight. Aggregate decode throughput at that concurrency is roughly 4x Super’s single-request throughput. Prefill of a 990K-token immediate is about 1.2x sooner.
How Iterative Puzzle Works
Puzzle is a decomposed neural structure search framework, carried out right here as Puzzletron. It defines a discrete search house of other layer implementations. Each various will get a high quality rating. A mixed-integer program then selects one various per layer beneath a deployment constraint.
Three pruning methods kind the search house:
- Intermediate channel pruning: Channels inside every routed professional are ranked by contribution to the professional’s output. All consultants inside one MoE layer are pruned to a uniform measurement, for kernel compatibility.
- Top-k discount: The variety of consultants a token is routed to varies per layer, as much as the guardian’s ok=22.
- Mamba SSM pruning: The SSM state measurement drops from 128 to 96 channels.
The SSM result’s measured. Dropping 128 channels to 96 speeds the SSM kernel 1.2x to 1.3x throughout decode. This holds at batch sizes between 8 and 512. Channels have been ranked by estimated contribution to the Mamba layer output. The estimate averaged over 67M tokens of validation information. Appendix A reveals this beats random channel choice beneath aggressive pruning.
The unique formulation assumes substitute high quality impacts are roughly additive. Each candidate block is scored contained in the unmodified guardian. That ignores higher-order interactions between replacements.
Iterative Puzzle alternates bounded compression with quick information distillation restoration. It builds a sequence M0, M1, … MR as an alternative of leaping to the goal. Scores are recomputed towards the present compressed mannequin, not the unique guardian.
Three levels have been used:
- MoE weights to 75% of instructor capability, Mamba SSM state to 75%. Healed for 24B tokens.
- MoE weights to 60% of instructor capability. Healed for 43.2B tokens.
- Activated routed-expert price range to 50%, allotted heterogeneously. Healed for 52.8B tokens.
The above desk compares this towards a single-step Puzzle baseline on the identical goal. The three-step process averages 69.05 throughout ten benchmarks, towards 68.48. Gains seem on MMLU-Pro, GPQA, HLE, AA-LCR, LiveCodeBench, SciCode, and RULER-256K. IFBench-Instruction fell 0.2 factors and IFBench-Prompt fell 0.5.
Recovery: Distillation, RL, and Verbosity
Knowledge distillation ran on 30% pretraining information and 70% SFT information from Nemotron-3-Nano. During the Puzzle part, KD used a 32K sequence size. Recovery then skilled at 128K, and scaled to 512K. The price range was as much as 100B tokens, with a 16M-token world batch, in Megatron-LM.
RL post-training adopted Stage 2 of the Nemotron-3-Super RL pipeline, centered on software program engineering. Phase 2.1 did single-step tool-use comparability. Phase 2.2 moved to end-to-end sandbox RL, the place brokers run as much as 200 turns. Both phases used a KL penalty of 0. The staff swept studying charges, then averaged the ensuing weights.
The above Figure 4 reveals what every stage contributed. Short-context KD recovers most classes to over 97% of Nemotron-3-Super. Long-context KD then lifts long-input and long-generation benchmarks particularly. The analysis staff states that RL’s influence in these experiments was small.
Verbosity is the quiet element. After the final Puzzle iteration, the mannequin generated 132% of Super’s token rely. That fell to 99% after the complete restoration pipeline.
Deployment: Quantization and Multi-Token Prediction
Two post-training quantization recipes have been produced: FP8 W8A8 targets Hopper and NVFP4 W4A4 targets Blackwell.
| Component | BF16 baseline | FP8 checkpoint | NVFP4 checkpoint |
|---|---|---|---|
| Sparse and shared MoE GEMMs | BF16 | FP8 | NVFP4 |
| Mamba GEMMs | BF16 | FP8 | FP8 |
| Mamba SSM cache | FP32 | FP32 | FP16+SR |
| KV cache | FP8 | FP8 | FP8 |
| Router | FP32 | FP32 | FP32 |
| Attention QKV/output, MoE latent projections, LM head | BF16 | BF16 | BF16 |
Both recipes calibrated on 256 post-training SFT samples. NVFP4 used max calibration, not the AutoQuantize sensitivity search used for Super. The ensuing checkpoint is barely extra aggressively quantized, and carried out equally.
NVFP4 will not be natively supported on Hopper. It remains to be used for the 1M-context H100 goal, as a result of HBM capability binds there.
Puzzle-75B-A9B inherits a shared MTP head from Super. Parameters are shared throughout MTP steps, so one head applies recursively at inference. Transferring Super’s skilled head instantly gave comparable acceptance lengths.
The analysis staff then identifies a training-inference mismatch. Teacher-forced MTP coaching feeds the complete shifted hidden-state sequence. Autoregressive drafting as an alternative feeds a mix of target-model and MTP-generated hidden states. Acceptance charges fall at deeper draft positions.
Continued coaching on the transferred head addresses this. On SPEED-Bench at draft size 7, common acceptance size rose from 3.45 to 4.34. That is roughly 25% to 30%, concentrated at later draft positions. Unlike Super, the NVFP4 checkpoint barely degrades: 4.31 towards 4.34.
Where Compression Helps and Where It Hurts
| Benchmark (BF16) | Super | Puzzle-75B-A9B | Delta |
|---|---|---|---|
| MMLU-Pro | 83.8 | 82.4 | -1.4 |
| AIME25 (no instruments) | 92.2 | 89.7 | -2.5 |
| GPQA (no instruments) | 80.5 | 78.6 | -1.9 |
| LiveCodeBench | 82.1 | 81.1 | -1.0 |
| SciCode (subtask) | 42.3 | 40.6 | -1.7 |
| SWE-Bench (OpenHands) | 59.5 | 56.9 | -2.6 |
| Arena-Hard-V2 | 72.8 | 68.6 | -4.2 |
| AA-LCR | 56.8 | 56.9 | +0.1 |
| RULER 1M | 93.9 | 92.2 | -1.7 |
| MMLU-ProX | 79.5 | 77.5 | -2.0 |
The analysis paper’s personal abstract is that instruction-following and agentic evaluations lose most. Arena-Hard-V2 is the worst case, at -4.2 factors. RULER stays inside roughly 1 to 2 factors at 256K, 512K, and 1M.
Three BF16 outcomes don’t regress. AA-LCR positive factors 0.1, Scale AI Multi-Challenge ties at 56.6, and TauBench Telecom positive factors 0.4.
NVFP4 prices little on high of compression. On RULER 1M the NVFP4 checkpoint scores 93.2, above BF16’s 92.2. HLE is the clearest NVFP4 price, dropping from 16.5 to fifteen.7. FP8 outcomes sit in Appendix E, and observe BF16 intently. SWE-Bench will not be reported for the FP8 checkpoint.
Use Cases
- Ultra-long-context RAG on one GPU: A doc evaluation service at 1M context goes from 1 concurrent request to eight. Aggregate decode throughput at that concurrency is roughly 4x.
- Interactive coding assistants: At UT >= 100 tok/s within the 8K/64K regime, one node serves 2.03x the tokens. Adjusted for verbosity, that’s 2.16x the finished requests per minute.
- Prefill-heavy doc pipelines: The 50K/2K regime positive factors only one.60x. Compression helps much less when immediate processing dominates compute.
- Agentic SWE loops: Check the two.6-point SWE-Bench hole towards your process combine. RL restoration focused this functionality, and solely partly restored it.
Deployment Explorer
Check out the Paper and Model weight. Also, be at liberty to comply with us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so forth.? Connect with us
The put up Meet Nemotron Labs 3 Puzzle 75B A9B: A Compressed Hybrid MoE LLM Delivering 2.03x Server Throughput appeared first on MarkTechPost.
