|

Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize

ByRicardo

An agent harness is the code round a mannequin: execution loop, instruments, context, state, restoration, and verification. Per the Terminal-Bench 2.1 leaderboard, GPT-5 solves 35.2% of duties inside Terminus 2 however 49.6% inside Codex CLI with similar weights. Most benchmarks preserve that harness fastened. HarnessDev proposed by staff of researchers from ByteDance Seed, Singapore University of Technology and Design, Georgia Institute of Technology, M-A-P, and TokenWave.AI, flips the goal: the artifact below analysis is the runnable harness the mannequin writes, not the reply it produces.

2 levels: Creation and Evolution

In Creation, each creator receives the identical weak seed: passive file, search, and course of primitives plus outcome and trajectory writers, with no loop, planner, verifier, retry, or stopping rule. Unmodified, it scores 0 in every single place. The creator will get a task-family spec, a brief design tutorial, and 1 to three improvement instances, builds a full harness, and the harness is frozen earlier than hidden duties.

In Evolution, the creator begins from its personal frozen Creation code harness and revises it utilizing execution suggestions from a set set of 100 SWE-bench Pro duties and all 89 Terminal-Bench 2.1 duties. Each official candidate should full each evaluations as a pair, with a funds of 10 pairs and at most 2 five-task probes between pairs. Every official model is later scored on 630 held-out SWE-Pro situations the creator by no means sees.

Harnesses are graded on functionality (activity success) and effectivity (executor tokens, with creator tokens excluded).

Setup

6 creator LLMs have been examined: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro, working inside Claude Code 2.1.177 (GPT-5.5 used Codex 0.144.3). Creation spans 4 domains and 5 benchmarks totaling 2,207 situations: SWE-bench Pro public break up (731), Terminal-Bench 2.1 (89), (*34*) (75), EQ-Bench3 (46), and BrowseComp (1,266). Each creator builds 3 harnesses per benchmark, reported as avg@3. Self-Eval runs every harness with its creator; Unified-Eval runs all with Gemini 3.1 Pro.

Creation outcomes

Under Self-Eval, Opus 4.8 posts the best common rating at 67.8 towards a human-engineered reference of 86.2. The hole is determined by area:

  • Code: Opus 4.8 reaches 69.3 on SWE-Pro versus the 80.0 reference. Gemini 3.1 Pro leads Terminal-Bench at 68.8 versus 88.8.
  • Search: the widest hole. The greatest BrowseComp rating is 52.6 (GPT-5.5) towards a 92.2 reference.
  • Writing: Opus 4.8 scores 84.6 on EQ-Bench3, above the 83.7 reference.
  • ML experimentation: Opus 4.8 (32.9) and Gemini (32.4) beat the 24.0 MLE-bench reference.

The SWE-Pro, Terminal-Bench, and BrowseComp references are exterior outcomes from OpenAI’s GPT-5.6 report, not re-runs.

Code quantity didn’t predict high quality: the 18 code harnesses added 17,111 web traces, but Gemini added the fewest (1,006) and led Terminal-Bench. Self-test depend barely correlated with rating (Spearman 0.13 to 0.26); revision calls reached 0.57.

Much generated equipment is inert. Of 108 code part situations, 72 set off in actual runs and 18 by no means fireplace, all of them state and reminiscence. 11 of 18 harnesses outline a State class, but no checkpoint occasion seems throughout 26,679 trajectories. 124 of 587 writing options are useless code.

Cost and executor switch

MLE-bench token use assorted roughly 19-fold. GPT-5.5 hit a 19.1 medal price with 29.3M tokens whereas DeepSeek V4 hit 19.6 with 208.4M. Swapping the executor to Gemini reshuffled rankings: Qwen gained 17.6 factors on BrowseComp and 12.9 on MLE-bench, whereas Opus 4.8’s SWE-Pro rating fell from 69.3 to 33.0, partly as a result of one harness hard-coded a 120-step restrict round its unique executor. The Opus search harness’s duplicate-query price jumped from 10.1% to 88.2% after the change.

Evolution outcomes

9 lineages (5 self-runtime, 4 fixed-Gemini) produced 73 official variations and 64 adjoining switches. All 5 self-runtime creators improved on held-out duties, from +1.43 to +4.44 factors (imply +3.11). Under fastened Gemini, solely Opus improved; GPT-5.5 regressed 10.32 factors.

Progress was not monotonic. Of 64 switches, 8 regressed on each benchmarks, 16 on one, 27 gained solely throughout the noise band, and a pair of confirmed clear optimistic proof. A single commit can differ by about ±4.75 pair-score factors. Feedback and held-out scores moved in the identical path solely 34 of 64 instances (53.1%), and solely 2 of 9 declared last variations have been held-out optimum. Of 169 new capabilities or courses, 25 haven’t any caller.

The clearest win: Opus 4.8 observed 99 of 100 runs reported success whereas solely 48 handed, traced it to untimely completion, and added a completion gate. Failure analysis was in any other case the weakest step: the devoted trajectory interface was referred to as solely twice.

Interactive explainer

Key Takeaways

  • HarnessDev scores the harness a mannequin builds, not the reply it returns.
  • Self-built harnesses match or beat references on writing and ML experimentation however path badly on code and search.
  • Harness high quality is executor-specific; Opus 4.8 drops from 69.3 to 33.0 on SWE-Pro below Gemini.
  • Evolution features are small, noisy, and solely 34 of 64 adjustments level the identical approach on held-out duties.
  • Much generated state and reminiscence code by no means executes.

Check out the Paper and Project Page. Also, be happy to observe us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The put up Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize appeared first on MarkTechPost.

Similar Posts