NVIDIA AI Releases Molt: A PyTorch-Native Agentic Reinforcement Learning Framework

Agentic reinforcement studying analysis is fixed algorithm modification. New estimators, new pipeline phases, new rollout schemes. In mainstream frameworks every change threads via layers of coach, distributed backend, and rollout glue. That price lands on the researcher at each iteration.

Molt, from NVIDIA’s NeMo staff, targets that price straight. Its a PyTorch-native agentic RL framework with an uncommon design goal. The codebase ought to be compact sufficient for a researcher to carry of their head, and for an AI coding assistant to learn and motive about in its entirety. The acknowledged footprint is roughly 8.6K traces of RL code, measured by tracing the import graph from every framework’s RL entry level. The similar methodology counts about 62K traces for verl, 25K for slime, and seven.2K for OpenRLHF.

Is it deployable?

Yes. Molt ships below Apache 2.0 with launch codes, Slurm scripts, and a prebuilt container. But the analysis paper positions it as analysis infrastructure, not a manufacturing coaching service, and {hardware} is the actual gate. The shipped recipes assume 2 nodes of 8 H100 GPUs, break up 8 for coaching and eight for rollout.

That places Molt in attain of frontier and frontier-adjacent labs, well-funded AI startups doing post-training, enterprise AI analysis teams in finance, healthcare, and robotics that prepare brokers in opposition to proprietary environments, and educational labs with multi-node H100/H200 entry. Applications embrace multi-turn tool-use brokers, code-execution brokers, vision-language environments (the shipped geo3k recipe), LLM-as-judge reward loops, and on-policy distillation onto a smaller scholar.

Three parts, one loop

Molt composes Ray for placement and asynchronous queues, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for coaching. None of the three is forked, so upstream enhancements arrive as a container pin reasonably than a rebase.

The runtime is an agent pool, a set of vLLM engines behind a request router, and a single trainable coverage actor. A streaming pool retains immediate teams in flight so engines by no means drain whereas the actor trains. Partial rollout pauses the engines, broadcasts actor shards over NCCL straight to every engine, and resumes retained requests as a substitute of discarding them.

The agent is an odd program

An RL run names one Python module that exports an AgentRunner. Everything else is odd code, together with the reward.

Two types are supported. With Env, the framework owns the LLM loop in a Gymnasium-aligned step(). With ChatAgent, the person owns the loop via a inventory OpenAI or Anthropic SDK. Molt launches a loopback server that speaks each wire protocols, and each request decodes server-side into one token-exact accumulation. When a long-horizon agent compacts its context and rewrites the prefix, the server seals the present phase and opens a contemporary one robotically.

Never prepare on a token you didn’t generate

Three correctness invariants arrange the design. Token identification: sampled token ids outline the trajectory, not a retokenized transcript. Policy-version semantics: trainable tokens hold their behavior-policy log-probabilities, and asynchronous use is corrected per token behind a sequence-level gate. Forward consistency: rollout and actor should agree on mannequin semantics.

For mixture-of-experts insurance policies, the final invariant issues most. The rollout and coaching routers choose specialists independently, and small numerical variations can flip top-k decisions. Molt applies rollout routing replay, the place vLLM returns its per-token professional ids and the coaching ahead replays them.

Interactive explainer