Gelato-30B-A3B: A State-of-the-Art Grounding Model for GUI Computer-Use Tasks, Surpassing Computer Grounding Models like GTA1-32B
How can we educate AI brokers to reliably discover and click on the precise on display component we imply after we give them a easy instruction? A workforce of researchers from ML Foundations has launched Gelato-30B-A3B, a state-of-the-art grounding mannequin for graphical consumer interfaces that’s designed to plug into laptop use brokers and convert pure language directions into dependable click on places. The mannequin is skilled on the Click 100k dataset and reaches 63.88% accuracy on ScreenSpot Pro and 69.15% on OS-World-G, with 74.65% on OS-World-G Refined. It surpasses GTA1-32B and bigger imaginative and prescient language fashions akin to Qwen3-VL-235B-A22B-Instruct.
What Gelato 30B A3B Does in An Agent Stack?
Gelato-30B-A3B is a 31B parameter mannequin that high-quality tunes Qwen3-VL-30B-A3B Instruct with a mix of specialists structure. It takes a screenshot and a textual instruction as enter and produces a single click on coordinate as output.
The mannequin is positioned as a modular grounding element. A planner mannequin, for instance GPT 5 within the Gelato experiments, decides the subsequent excessive degree motion and calls Gelato to resolve that step right into a concrete click on on the display. This separation between planning and grounding is essential when an agent should function throughout many working methods and purposes with completely different layouts.
Click 100k, A Targeted Dataset For GUI Grounding
Click 100k is the dataset that underlies Gelato. It pairs laptop display photographs with pure language directions, bounding bins for the goal component, picture dimensions, and normalized bounding bins. Each pattern is ready up as a low degree command, for instance ‘faucet on the component between Background and Notifications choices’ with a exact area.
The dataset is constructed by filtering and unifying a number of public sources. The record contains ShowUI, AutoGUI, PC Agent E, WaveUI, OS Atlas, UGround, PixMo Points, SeeClick, UI VISION, a JEDI subset that focuses on spreadsheet and textual content cell manipulation, and movies from 85 skilled software tutorials annotated with Claude-4-Sonnet. Each supply contributes at most 50k samples, and all sources are mapped right into a shared schema with photographs, directions, bounding bins, and normalized coordinates.
The analysis workforce then runs an aggressive filtering pipeline. OmniParser discards clicks that don’t land on detected interface parts. Qwen2.5-7B-VL and SE-GUI-3B take away trivial examples, akin to simple hyperlink clicks. GTA1-7B-2507 and UI-Venus-7B take away samples the place the instruction and click on area don’t match. A Qwen2.5-7B-VL baseline skilled on a balanced 10k subset exhibits that this mixture offers a +9 pp accuracy acquire on ScreenSpot Pro in contrast with coaching on unfiltered knowledge.
Professional software protection is a selected focus. Click 100k provides knowledge from UI VISION and the JEDI subset, after which augments this with 80+ tutorial movies for actual desktop instruments. Claude 4 Sonnet generates bounding bins and low degree directions for these movies, adopted by handbook inspection and corrections.
GRPO Training On Top Of Qwen3 VL
On the coaching facet, Gelato 30B A3B makes use of GRPO, a reinforcement studying algorithm that derives from work on DeepSeekMath and related methods. The analysis workforce comply with the DAPO setup. They take away the KL divergence time period from the target, set the clip larger threshold to 0.28, and skip rollouts with zero benefit. Rewards are sparse and are solely given when the expected click on falls contained in the goal bounding field, just like the GTA1 recipe.
They initialize from Qwen3 VL 30B A3B Instruct and run 100 RL steps on 32 A100 GPUs with 40 GB reminiscence. The finest checkpoint seems at step 84 (marked as inexperienced cross within the above picture), chosen by the imply efficiency throughout ScreenSpot Pro, OS World G, and OS World G Refined. At this level the mannequin reaches 63.88% on ScreenSpot-Pro and 67.19% and 73.40% on OS World G and OS World G Refined. A easy refusal prompting technique, which appends an instruction to reply with refusal when the component can’t be discovered, raises the OS-World-G scores to 69.15% and 74.65%.
End To End Agent Results On OS World
To check Gelato past static grounding benchmarks, the analysis workforce plugs it into the GTA1.5 agent framework and runs full laptop use brokers on the OS World surroundings. In this setup GPT 5 acts because the planner. Gelato 30B A3B offers grounding, the agent has at most 50 steps, and it waits 3 seconds between actions.
The analysis experiences three runs per mannequin on a hard and fast OS World snapshot. Gelato-30B-A3B reaches 58.71% automated success charge with a small normal deviation, in contrast with 56.97% for GTA1 32B in the identical harness. Because the automated OS World analysis misses some legitimate options, additionally they run human analysis on 20 problematic duties. Under human scoring, Gelato reaches 61.85% success, whereas GTA1-32B reaches 59.47%.
Key Takeaways
- Gelato-30B-A3B is a Qwen3-VL-30B-A3B Instruct based mostly combination of specialists mannequin that performs state-of-the-art GUI grounding on ScreenSpot Pro and OS World G benchmarks, surpassing GTA1-32B and bigger VLMs akin to Qwen3-VL-235B-A22B-Instruct.
- The mannequin is skilled on Click 100k, a curated grounding dataset that merges and filters a number of public GUI datasets {and professional} software traces, pairing actual screens with low degree pure language instructions and exact click on coordinates.
- Gelato-30B-A3B makes use of a GRPO reinforcement studying recipe on prime of Qwen3-VL, with sparse rewards that solely set off when the expected click on lies inside the bottom fact bounding field, which considerably boosts grounding accuracy over supervised baselines.
- When built-in into an agent framework with GPT-5 appearing because the planner, Gelato-30B-A3B improves success charges on OS World laptop use duties in contrast with GTA1-32B, demonstrating that higher grounding straight interprets into stronger finish to finish agent efficiency.
Editorial Comments
Gelato-30B-A3B is a vital step for grounded laptop use as a result of it exhibits {that a} Qwen3-VL based mostly MoE mannequin, skilled on a rigorously filtered Click 100k dataset, can beat each GTA1-32B and far bigger VLMs like Qwen3-VL-235B-A22B Instruct on ScreenSpot Pro and OS-World-G whereas staying accessible via Hugging Face. Overall, Gelato-30B-A3B establishes a transparent new baseline for open laptop grounding fashions.
Check out the Repo and Model Weights. Feel free to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Also, be at liberty to comply with us on Twitter and don’t overlook to hitch our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The put up Gelato-30B-A3B: A State-of-the-Art Grounding Model for GUI Computer-Use Tasks, Surpassing Computer Grounding Models like GTA1-32B appeared first on MarkTechPost.
