Google AI Unveils Supervised Reinforcement Learning (SRL): A Step Wise Framework with Expert Trajectories to Teach Small Language Models to Reason through Hard Problems

How can a small mannequin study to clear up duties it at the moment fails at, with out rote imitation or counting on an accurate rollout? A crew of researchers from Google Cloud AI Research and UCLA have launched a coaching framework, ‘Supervised Reinforcement Learning’ (SRL), that makes 7B scale fashions truly study from very exhausting math and agent trajectories that ordinary supervised high-quality tuning and consequence based mostly reinforcement studying RL can not study from.

Small open supply fashions similar to Qwen2.5 7B Instruct fail on the toughest issues in s1K 1.1, even when the instructor hint is sweet. If we apply supervised high-quality tuning on the complete DeepSeek R1 type options, the mannequin imitates token by token, the sequence is lengthy, the information is only one,000 objects, and the ultimate scores drop under the bottom mannequin.

Core concept of ‘Supervised Reinforcement Learning’ SRL

‘Supervised Reinforcement Learning’ (SRL) retains the RL type optimization, however it injects supervision into the reward channel as a substitute of into the loss. Each skilled trajectory from s1K 1.1 is parsed right into a sequence of actions. For each prefix of that sequence, the analysis crew creates a brand new coaching instance, the mannequin first produces a non-public reasoning span wrapped in <assume> … </assume>, then it outputs the motion for that step, and solely this motion is in contrast with the instructor motion utilizing a sequence similarity metric based mostly on difflib. The reward is dense as a result of each step has a rating, even when the ultimate reply is improper. The remainder of the textual content, the reasoning half, isn’t constrained, so the mannequin can search its personal chain with out being pressured to copy the instructor tokens.

Math outcomes

All fashions are initialized from Qwen2.5 7B Instruct and all are educated on the identical DeepSeek R1 formatted s1K 1.1 set, so comparisons are clear. The precise numbers in Table 1 are:

• Base Qwen2.5 7B Instruct, AMC23 grasping 50.0, AIME24 grasping 13.3, AIME25 grasping 6.7.
• SRL, AMC23 grasping 50.0, AIME24 grasping 16.7, AIME25 grasping 13.3.
• SRL then RLVR, AMC23 grasping 57.5, AIME24 grasping 20.0, AIME25 grasping 10.0.

This is the important thing enchancment, SRL alone already removes the SFT degradation and raises AIME24 and AIME25, and when RLVR is run after SRL, the system reaches the very best open supply scores within the analysis. The analysis crew is express that the very best pipeline is SRL then RLVR, not SRL in isolation.

Software engineering outcomes

The analysis crew additionally applies SRL to Qwen2.5 Coder 7B Instruct utilizing 5,000 verified agent trajectories generated by claude 3 7 sonnet, each trajectory is decomposed into step sensible situations, and in complete 134,000 step objects are produced. Evaluation is on SWE Bench Verified. The base mannequin will get 5.8 p.c within the oracle file edit mode and three.2 p.c finish to finish. SWE Gym 7B will get 8.4 p.c and 4.2 p.c. SRL will get 14.8 p.c and eight.6 p.c, which is about 2 instances the bottom mannequin and clearly greater than the SFT baseline.

Key Takeaways

1. SRL reformulates exhausting reasoning as step sensible motion technology, the mannequin first produces an inside monologue then outputs a single motion, and solely that motion is rewarded by sequence similarity, so the mannequin will get sign even when the ultimate reply is improper.
2. SRL is run on the identical DeepSeek R1 formatted s1K 1.1 information as SFT and RLVR, however in contrast to SFT it doesn’t overfit lengthy demonstrations, and in contrast to RLVR it doesn’t collapse when no rollout is right.
3. On math, the precise order that provides the strongest ends in the analysis is, initialize Qwen2.5 7B Instruct with SRL, then apply RLVR, which pushes reasoning benchmarks greater than both methodology alone.
4. The identical SRL recipe generalizes to agentic software program engineering, utilizing 5,000 verified trajectories from claude 3 7 sonnet 20250219, and it lifts SWE Bench Verified effectively above each the bottom Qwen2.5 Coder 7B Instruct and the SFT type SWE Gym 7B baseline.
5. Compared to different step sensible RL strategies that want an additional reward mannequin, this SRL retains a GRPO type goal and makes use of solely actions from skilled trajectories and a light-weight string similarity, so it’s straightforward to run on small exhausting datasets.

Editorial Comments

‘Supervised Reinforcement Learning’ (SRL) is a sensible contribution by the analysis crew. It retains the GRPO type reinforcement studying setup, however it replaces fragile consequence stage rewards with supervised, step sensible rewards which are computed straight from skilled trajectories, so the mannequin at all times receives informative sign, even within the D_exhausting regime the place RLVR and SFT each stall. It is essential that the analysis crew exhibits SRL on math and on SWE Bench Verified with the identical recipe, and that the strongest configuration is SRL adopted by RLVR, not both one alone. This makes SRL a sensible path for open fashions to study exhausting duties. Overall, SRL is a clear bridge between course of supervision and RL that open mannequin groups can undertake instantly.

---

Check out the Paper. Feel free to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Also, be at liberty to comply with us on Twitter and don’t overlook to be part of our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

The publish Google AI Unveils Supervised Reinforcement Learning (SRL): A Step Wise Framework with Expert Trajectories to Teach Small Language Models to Reason through Hard Problems appeared first on MarkTechPost.