|

Cursor Open-Sources Mixture-of-Kittens (MoK): A Deterministic MoE Training Megakernel for GB300 NVL72 Racks

ByRicardo

Cursor Research has open-sourced Mixture-of-Kittens (MoK), the mixture-of-experts coaching megakernel behind its Composer fashions. MoK fuses each MoE communication and computation step right into a single deterministic kernel. Cursor group studies as much as 2.37x greater throughput than the strongest public baseline. It already powers Composer coaching throughout tens of hundreds of GPUs.

Is it deployable

Yes, however the {hardware} flooring is excessive. MoK is on GitHub underneath Apache-2.0. It requires NVIDIA Blackwell SM100 or SM103 GPUs, which suggests GB200 NVL72 or GB300 NVL72 racks. It additionally wants Python 3.12+, PyTorch 2.10+, and CUDA toolkit 13.0+. Inter-GPU buffers depend on PyTorch symmetric reminiscence.

That limits reasonable adopters to organizations that personal or hire NVL72 capability. Frontier labs, funded mannequin startups, GPU neoclouds, and nationwide computing facilities match. Single-node groups and 8-GPU outlets don’t.

Applications are slim however high-value. They embrace pretraining and post-training of DeepSeek-V3-style MoE fashions. Determinism additionally makes it helpful for on-policy RL post-training and inner ablations. Relevant industries are AI mannequin improvement, cloud GPU infrastructure, code-generation tooling, and quantitative analysis.

MoE layer because the bottleneck

Cursor’s earlier work lined the compute aspect. The analysis group wrote its personal MXFP8 and NVFP4 training kernels and a ‘warp decode’ path for MoE inference. Those assumed inter-GPU communication was dealt with individually.

In manufacturing, communication grew to become the limiting issue. The MoE layer can eat greater than half of end-to-end coaching time. Moving to GB300 NVL72s modified the issue once more. A rack is 72 GPUs inside one NVLink area, which permits fine-grained overlap. But the built-in Grace CPUs are gradual relative to the GPUs. CPU-GPU synchronization due to this fact must be minimized aggressively.

Three design choices that matter

  • Communication path is chosen per operation: Existing approaches reminiscent of DeepEP lean on push-based transfers. Cursor’s microbenchmarks present push strikes fewer complete bytes in a single path. That leaves the reverse NVLink lane largely idle. Pull-based dispatch delivers as much as 29% greater NVLink bandwidth utilization underneath knowledgeable imbalance. It additionally eliminates cross-GPU completion indicators. Push dispatch signalling measured 103 µs in opposition to 18 µs for pull, roughly 5.8x. MoK due to this fact makes use of pull-based ahead dispatch and push-based ahead mix. The backward move mirrors this with pull reverse-combine and push reverse-dispatch. One schedule desk serves all 4, costing underneath 3% of MoE runtime.
  • Overlap granularity sits between the extremes: Comet is fine-grained; DeepEP is coarse-grained. Cursor group argues the optimum is within the center and workload-dependent. The heuristic targets at the very least two full SM waves per expert-grouped GEMM. For Kimi 2.5 shapes, the bottom mannequin for Composer 2.5, the ground is 2,368 tokens. Measured latency matches that estimate carefully.
  • A ring token buffer removes the CPU from the loop: The options are dropping tokens or asking the CPU to measurement buffers. MoK as an alternative cycles a set ring buffer of some hundred megabytes. It does so at minibatch granularity, interleaving dispatch and mix at macrobatch boundaries. The ring is walked in reverse to reduce ahead activation replay throughout backward.

MoK is constructed as a megakernel and is totally deterministic. It helps BF16 and MXFP8 precision modes. Scheduling runs by Blackwell’s Cluster Launch Control, so inter-rack RDMA doesn’t serialize behind it. Router weight gradients use a SonicMoE-style calculation fused into the SwiGLU backward.

https://cursor.com/weblog/mixture-of-kittens

Results

Layer benchmarks ran in a single NVL72 rack at EP diploma 64. Each GPU held 2,048 tokens earlier than routing. Baselines had been NCCL+PyTorch, DeepEP+PyTorch, DeepEP+TransformerEngine, and HybridEP+Megatron. Shapes lined Kimi K2.7 Code, GLM-5.2, Qwen3.5-397B-A17B, and DeepSeek-V4-Pro.

Against the quickest baseline, MoK is as much as 2.37x sooner for MXFP8 ahead. The different figures are 1.78x MXFP8 backward, 1.92x BF16 ahead, and 1.58x BF16 backward. End-to-end testing used 512 GPUs throughout a number of GB300 NVL72 racks. Tokens per second per GPU rose from 760.9 to 1,070.2, a 1.41x achieve.



Key Takeaways

  • MoK fuses all MoE communication and computation into one deterministic megakernel for NVL72 racks.
  • Pull dispatch plus push mix cuts signalling from 103 µs to 18 µs.
  • A ring token buffer drops zero tokens and removes CPU-GPU synchronization fully.
  • Up to 2.37x over the quickest public baseline; 1.41x end-to-end on 512 GPUs.
  • Apache-2.0, however it calls for Blackwell SM100/SM103, CUDA 13.0+, and PyTorch 2.10+.

Check out the GitHub Repo and Technical details. Also, be happy to comply with us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so forth.? Connect with us

The publish Cursor Open-Sources Mixture-of-Kittens (MoK): A Deterministic MoE Training Megakernel for GB300 NVL72 Racks appeared first on MarkTechPost.

Similar Posts