Meta FAIR Introduces AI Research Preference Models (RPMs): Ranking ML Experiments Before Spending GPU Hours
AI analysis brokers can already suggest, implement and rating their very own machine studying experiments. Idea technology is reasonable; verification isn’t. Training one candidate can eat hours to days of GPU time, so an agent proposes way more candidates than it will probably afford to run. Which ones get run is the actual lever on analysis progress.
A analysis crew from FAIR at Meta, the University of Oxford and University College London formalizes that lever as analysis desire and introduces AI Research Preference Models (RPMs). An RPM ranks unexecuted candidates and picks one to execute. It by no means forecasts an absolute rating, the crew discovered language fashions unreliable at predicting metrics or execution outcomes.
Is it deployable? Partially. RPMs use frozen pretrained LLMs with no fine-tuning, the scaffold AIRA-dojo and benchmark AIRS-Bench are open supply, and the spine Qwen3.6-27B is open weights.
Where the RPM sits within the agent loop
AIRA-dojo is an evolutionary tree search: grasping dad or mum choice, Draft / Improve / Debug operators, highest-validation-score node returned on the finish. The RPM intervenes at little one creation solely. Instead of producing one little one and executing it, the agent applies the operator 15 instances in parallel to yield 15 unexecuted candidates, then compares them pairwise in a knockout event. Only the winner is executed. Each comparability is grounded in context nodes collected by a BFS stroll of the explored tree, every proven with the validation rating it obtained.