Guide to Loop Engineering: How ‘autoresearch’ and ‘Bilevel Autoresearch’ Turn AI Agents Into Autonomous Machine Learning ML Research Loops

Most folks nonetheless use AI like a 2015 search field. You kind, you learn, you kind once more. A more moderen sample replaces that handbook back-and-forth with a loop. This information explains loop engineering utilizing two verified artifacts. The sources are Andrej Karpathy’s autoresearch repository and the Bilevel Autoresearch paper. The framing follows a write-up by @0xCodila.

What is Loop Engineering?

To begin, examine two modes. A immediate is one instruction, after which you resolve the subsequent step. A loop, against this, is a objective the mannequin pursues till it arrives. The mannequin plans, acts, checks its personal end result, then repeats. You outline the target as soon as, and the loop handles iteration. Crucially, a loop solely earns its value when the work is measurable.

The Three Parts That Make A Loop Work

So what separates an actual loop from a chatbot on repeat? Every dependable loop has three elements.

1. A verifier grades every try. That examine is usually a passing take a look at, a transferring metric, or a construct. Without a verifier, the agent merely agrees with itself on repeat.
2. State data what was tried, what failed, and what stays. A small aspect file lets the subsequent run resume as an alternative of restarting.
3. A cease situation prevents runaway value. The loop halts when the objective is met, or after N makes an attempt.

The Karpathy Loop: Inside ‘autoresearch’

These three components usually are not theoretical. On March 7, 2026, Karpathy launched autoresearch, an open-source repository underneath the MIT license. It ships three core information and about 630 strains of code. The venture went viral inside days and now sits close to 90,000 GitHub stars. It was latter introduced because the sample “the Karpathy Loop.”

The design is intentionally small, but strict. The agent edits solely practice.py, which holds the GPT mannequin, optimizer (Muon and AdamW), and coaching loop. It can not contact the analysis utilities in put together.py. That separation stops the agent from making the take a look at simpler as an alternative of the mannequin higher. Meanwhile, a human writes program.md, the directions the agent should respect.

Each cycle runs one experiment. The agent reads the code, proposes a change, trains 5 minutes, then retains or rolls again. The scoring metric is val_bpb, validation bits per byte, the place decrease is healthier. That price range yields roughly 12 experiments per hour, so about 100 run in a single day.

The reported outcomes are concrete. Karpathy pointed it at his already-optimized nanochat GPT-2 coaching code. It ran for 2 days and accomplished about 700 experiments, holding 20 real enhancements. Stacked collectively, these fixes minimize GPT-2-quality coaching time by 11%, from 2.02 to 1.80 hours. One discovering was a QK-Norm implementation lacking a scalar multiplier, which had left consideration too diffuse throughout heads.

Notably, people tire after roughly a dozen experiments, whereas the loop doesn’t. Separately, Shopify CEO Tobi Lütke ran autoresearch in a single day on an inner mannequin. He reported a 19% enchancment after 37 experiments. Karpathy’s takeaway: you probably have an goal metric, you’re the bottleneck.

Prompt vs Loop vs Bilevel Loop

The variations grow to be clearer aspect by aspect.

The Five Building Blocks

Consequently, AI engineering groups now assemble working loops from 5 reusable items:

• Automation fires the loop on a schedule, occasion, or set off.
• A talent shops venture data in a markdown file, learn on each run.
• Sub-agents cut up the author from the reviewer, since one mannequin grades itself too generously.
• Connectors let the loop act inside actual instruments, like a problem tracker or Slack.
• Finally, a verifier stays the gate that rejects unhealthy work. Claude Code and Codex now ship all 5.

Bilevel Autoresearch: A Loop On Top Of The Loop

Next, researchers requested a sharper query. If autoresearch is analysis, are you able to autoresearch autoresearch? The analysis paper Bilevel Autoresearch: Meta-Autoresearching Itself solutions sure.

The interior loop matches Karpathy’s unique: suggest, practice, consider, maintain or discard. The outer loop watches the interior loop and reads its code and traces. It identifies the place the search itself retains stalling. Then it writes new Python mechanisms, injects them at runtime, and reruns the interior loop.

The end result held on Karpathy’s GPT pretraining benchmark. The outer loop minimize val_bpb 5x greater than the one loop (-0.045 vs -0.009). Notably, each loops used the identical LLM, so the acquire got here from structure, not a better mannequin. In follow the design splits into three ranges. Level 1 runs the bottom loop. Level 1.5 tunes search parameters each 5 iterations. Level 2 generates mechanisms by way of a four-round session. The reported experiments used an RTX 5090 32GB and a 300-second price range.

The cause is price noting. The interior loop saved returning to the identical priors, even after they stopped working. The outer loop broke these patterns by forcing unfamiliar exploration.

Use Cases With Examples

These concepts switch properly past pretraining. For mannequin work, a loop searches hyperparameters till val_bpb drops. For software program, it refactors till checks, sorts, and the construct move. For content material, it rewrites till each rubric rating clears a threshold. For information, it tunes a pipeline till schema checks maintain. Each case shares one trait: an computerized gate that may fail the work.

Try It Yourself: A Loop In One Prompt

Theory apart, you’ll be able to really feel the mechanic with out Claude Code or Codex. Paste this into any succesful mannequin and watch it self-correct.

You will work in a loop till the duty meets the bar.

TASK:
[describe exactly what you want produced]

SUCCESS CRITERIA (be strict):
- [criterion 1]
- [criterion 2]
- [criterion 3]

LOOP PROTOCOL, repeat each flip:
1. PLAN   - state the one subsequent step.
2. DO     - produce or enhance the work.
3. VERIFY - rating the end result 1-10 on every criterion. Be sincere.
4. DECIDE - if each criterion is 8+, print FINAL and cease.
            Otherwise print ITERATING and repair the weakest level first.

RULES:
- Never name it completed till each criterion is 8 or greater.
- Each move should repair the weakest rating from the final VERIFY.
- Do not ask questions. Make a smart assumption and proceed.
Begin.

Underneath, the management movement is small. The skeleton under reveals these three components in Python: a verifier, a call, and two cease circumstances.

present = baseline
greatest = consider(present)                 # verifier: decrease val_bpb is healthier
for step in vary(MAX_STEPS):            # cease situation 1: experiment price range
    candidate = propose_change(present)  # agent edits practice.py
    rating = train_and_eval(candidate)    # practice 5 min, then confirm
    if rating < greatest:                     # maintain solely actual enhancements
        present, greatest = candidate, rating # commit
    # else: discard candidate, restore baseline
    if greatest <= TARGET:                   # cease situation 2: objective met
        break

Both variations are restricted. You are nonetheless the set off, and closing the tab erases the state. Adding automation, a state file, and an actual gate turns this into an autonomous loop.

See It Run

The interactive demo under animates one full loop: suggest, practice, confirm, then maintain or roll again. Adjust the goal and step restrict, and watch val_bpb fall till the cease situation fires.