|

Meet ‘Code-as-World’: An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs

ByRicardo

MirroS launched Code-as-World: a paradigm that represents bodily worlds by means of executable world representations. The argument is slender and testable: pixels are proof of a bodily scene, not its ontology. A video mannequin can predict believable frames with out ever representing mass, contact, or gravity. So as an alternative of pixels, latents, or captions, Code-as-World represents a scene as executable code — a scene.json that MuJoCo can run, that an agent can confirm in opposition to the supply video, and that anybody can edit and re-simulate. An agentic loop recovers these packages from actual footage in as much as 5 rounds. The verified worlds then turn into coaching knowledge with actual bodily labels, which actual video doesn’t carry. Trained on that supervision, Code-as-World-VL-9B scores 55.4 MRA on QuantiPhy-validation, above Gemini-3.1 Flash at 54.8 and roughly 15 factors above the strongest open-weight baseline.

Is it deployable?

Yes, on the analysis and internal-prototype tier. MirroS shipped the GitHub repo and two checkpoints — Code-as-World-VL-4B and Code-as-World-VL-9B — below Apache 2.0, fine-tuned from Qwen3.5-4B and Qwen3.5-9B. Both are BF16 safetensors served by vLLM behind an OpenAI-compatible /v1 endpoint, with 16 sampled frames per video and --max-model-len 4608.


The concept: pixels are proof, not ontology

The MirroS technical report argues that video fashions, 3D reconstruction, and captions every recuperate a part of a scene however none recovers its mechanism. Code-as-World represents a scene as an executable world illustration (EWR), a triple p = (C, E, A):

  • Composition: objects, geometry, metric dimensions, mass, friction, gravity. Floors and partitions are static bodily entities to allow them to assist and collide.
  • Evolution: preliminary states, forces, contacts, collisions, termination circumstances, length. Executing it expands composition right into a full state trajectory.
  • Appearance: digicam, lighting, supplies, background, body price, render config. Changing it by no means modifications the physics.

In the launched implementation, that triple compiles right into a scene.json executed in MuJoCo, with two interchangeable engines: an animation engine (kinematic poses) and a physics engine (forces and contacts).

Agentic discovery as an alternative of one-shot prediction

Recovering an EWR from a video is an inverse drawback, so the crew frames it as abductive search. An agent runs suggest → instantiate → execute → render → confirm for as much as Ok = 5 rounds. For video enter, SAM 3 provides occasion masks and image-plane tracks, VGGT-Omega estimates depth and digicam geometry, and SAM 3D generates per-object meshes. Candidate rollouts are projected again into the enter view and in contrast at chosen key frames on RGB, depth, masks, and trajectories. Frame-level discrepancies mixture into structured suggestions Δ that guides the subsequent revision; when the price range runs out with out acceptance, the speculation is rejected.

At a matched five-evaluation price range, the loop beats Best-of-5 impartial sampling on Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2percentD — and the consequence repeats below the second execution engine. Candidate movies come from WISA-80K after motion-focused filtering; sim-to-real re-rendering makes use of Wan2.2-VACE plus an inner video mannequin.

Verified worlds as coaching supervision

Phase 1 is supervised fine-tuning on 73,335 image-space QA pairs constructed from RefCOCO/+/g, RefCLEF and GOT-10K, protecting extent, place, displacement, velocity and acceleration in uncooked pixels. Phase 2 applies GRPO to world-space VQA drawn from 1,585 text-driven and 988 video-driven executable worlds, rewarded on scale-normalized numerical accuracy plus unit and format phrases. Training used eight NVIDIA H100 GPUs.

On QuantiPhy-validation (159 objects, MRA macro-averaged over 2S/2D/3S/3D): 4B = 50.6, 9B = 55.4, 27B reasoning = 58.6, in opposition to Gemini-3.1 Flash at 54.8, ChatGPT-5.1 at 48.4, and the strongest open-weight baseline Qwen3-VL-32B-Instruct at 40.2. The ablation is the extra helpful quantity: image-space-only scores 44.2 (4B) and 50.9 (9B); including each world-space sources lifts them to 50.6 and 55.4. Pixel-level grounding improves too — the 9B goes 63.7 → 68.3 on RefCOCO and 20.1 → 26.6 on GOT-10K after world-space RL.

Key Takeaways

  • Code-as-World turns a video into an editable scene.json that MuJoCo can execute and confirm.
  • Five-round suggest→confirm search beats Best-of-5 sampling on the similar compute price range.
  • Verified worlds provide actual bodily labels that actual video merely doesn’t carry.
  • 9B hits 55.4 MRA on QuantiPhy, above Gemini-3.1 Flash at 54.8; 4B and 9B are Apache 2.0.
  • Rigid-body solely, and the mannequin by no means learns the invention loop itself.

Check out the Technical report, MirroS blog, Project page, GitHub and Announcement on X. Also, be happy to observe us on Twitter and don’t overlook to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The submit Meet ‘Code-as-World’: An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs appeared first on MarkTechPost.

Similar Posts