Sigmoidal Scaling Curves Make Reinforcement Learning RL Post-Training Predictable for LLMs

Reinforcement Learning RL post-training is now a significant lever for reasoning-centric LLMs, however not like pre-training, it hasn’t had predictive scaling guidelines. Teams pour tens of hundreds of GPU-hours into runs and not using a principled strategy to estimate whether or not a recipe will hold enhancing with extra compute. A brand new analysis from Meta, UT Austin, UCL, Berkeley, Harvard, and Periodic Labs supplies a compute-performance framework—validated over >400,000 GPU-hours—that fashions RL progress with a sigmoidal curve and provides a examined recipe, ScaleRL, that follows these predicted curves as much as 100,000 GPU-hours.

Fit a sigmoid, not an influence regulation

Pre-training usually matches energy legal guidelines (loss vs compute). RL fine-tuning targets bounded metrics (e.g., go price/imply reward). The analysis staff present sigmoidal matches to go price vs coaching compute are empirically extra strong and steady than power-law matches, particularly if you wish to extrapolate from smaller runs to bigger budgets. They exclude the very early, noisy regime (~first 1.5k GPU-hours) and match the predictable portion that follows. The sigmoidal parameters have intuitive roles: one units the asymptotic efficiency (ceiling), one other the effectivity/exponent, and one other the midpoint the place good points are quickest.

Why that issues: After ~1–2k GPU-hours, you’ll be able to match the curve and forecast whether or not pushing to 10k–100k GPU-hours is value it—earlier than you burn the price range. The analysis additionally reveals power-law matches can produce deceptive ceilings except you solely match at very excessive compute, which defeats the aim of early forecasting.

ScaleRL: a recipe that scales predictably

ScaleRL isn’t just new algorithm; it’s a composition of decisions that produced steady, extrapolatable scaling within the research:

• Asynchronous Pipeline RL (generator–coach cut up throughout GPUs) for off-policy throughput.
• CISPO (truncated importance-sampling REINFORCE) because the RL loss.
• FP32 precision on the logits to keep away from numeric mismatch between generator and coach.
• Prompt-level loss averaging and batch-level benefit normalization.
• Forced size interruptions to cap runaway traces.
• Zero-variance filtering (drop prompts that present no gradient sign).
• No-Positive-Resampling (take away high-pass-rate prompts ≥0.9 from later epochs).

The analysis staff validated every element with leave-one-out (LOO) ablations at 16k GPU-hours and present that ScaleRL’s fitted curves reliably extrapolate from 8k → 16k, then maintain at a lot bigger scales—together with a single run prolonged to 100k GPU-hours.

Results and generalization

Two key demonstrations:

1. Predictability at scale: For an 8B dense mannequin and a Llama-4 17B×16 MoE (“Scout”), the prolonged coaching intently adopted the sigmoid extrapolations derived from smaller-compute segments.
2. Downstream switch: Pass-rate enhancements on an iid validation set monitor downstream analysis (e.g., AIME-24), suggesting the compute-performance curve isn’t a dataset artifact.

The analysis additionally compares fitted curves for prevalent recipes (e.g., DeepSeek (GRPO), Qwen-2.5 (DAPO), Magistral, MiniMax-M1) and experiences increased asymptotic efficiency and higher compute effectivity for ScaleRL of their setup.

Which knobs transfer the ceiling vs the effectivity?

The framework enables you to classify design decisions:

• Ceiling movers (asymptote): scaling mannequin measurement (e.g., MoE) and longer era lengths (as much as 32,768 tokens) increase the asymptotic efficiency however could gradual early progress. Larger international batch measurement can even carry the ultimate asymptote and stabilize coaching.
• Efficiency shapers: loss aggregation, benefit normalization, information curriculum, and the off-policy pipeline primarily change how briskly you method the ceiling, not the ceiling itself.

Operationally, the analysis staff advises becoming curves early and prioritizing interventions that increase the ceiling, then tune the effectivity knobs to achieve it sooner at mounted compute.

Key Takeaways

• The analysis staff fashions RL post-training progress with sigmoidal compute-performance curves (pass-rate vs. log compute), enabling dependable extrapolation—not like power-law matches on bounded metrics.
• A best-practice recipe, ScaleRL, combines PipelineRL-k (asynchronous generator–coach), CISPO loss, FP32 logits, prompt-level aggregation, benefit normalization, interruption-based size management, zero-variance filtering, and no-positive-resampling.
• Using these matches, the analysis staff predicted and matched prolonged runs as much as 100k GPU-hours (8B dense) and ~50k GPU-hours (17B×16 MoE “Scout”) on validation curves.
• Ablations present some decisions transfer the asymptotic ceiling (A) (e.g., mannequin scale, longer era lengths, bigger international batch), whereas others primarily enhance compute effectivity (B) (e.g., aggregation/normalization, curriculum, off-policy pipeline).
• The framework supplies early forecasting to determine whether or not to scale a run, and enhancements on the in-distribution validation monitor downstream metrics (e.g., AIME-24), supporting exterior validity.

Editorial Comments

This work turns RL post-training from trial-and-error into forecastable engineering. It matches sigmoidal compute-performance curves (pass-rate vs. log compute) to foretell returns and determine when to cease or scale. It additionally supplies a concrete recipe, ScaleRL, that makes use of PipelineRL-style asynchronous era/coaching, the CISPO loss, and FP32 logits for stability. The research experiences >400,000 GPU-hours of experiments and a single-run extension to 100,000 GPU-hours. Results assist a clear cut up: some decisions increase the asymptote; others primarily enhance compute effectivity. That separation helps groups prioritize ceiling-moving adjustments earlier than tuning throughput knobs.

---

Check out the PAPER. Feel free to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Also, be happy to comply with us on Twitter and don’t overlook to hitch our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

The publish Sigmoidal Scaling Curves Make Reinforcement Learning RL Post-Training Predictable for LLMs appeared first on MarkTechPost.