Meet LingBot-World-Infinity: An Open Causal World Model With An Agentic Harness

Robbyant, Ant Group’s embodied-intelligence unit, has launched LingBot-World-Infinity (LingBot-World 2.0). It is a causal video technology mannequin that behaves as an interactive world simulator. It is how the group assaults two failure modes: long-horizon drift and interactive latency.

What is LingBot-World-Infinity?

An interactive world model generates video frame by frame, conditioned on a stream of consumer actions. Each state relies upon solely on previous frames and present enter. The analysis group formalizes this as a causal factorization:

p_θ(x_1:T | a_1:T) = Π_t  p_θ(x_t | x_<t, a_≤t)

Here x_t is the visible state at time t. The motion a_t combines a digital camera pose and a textual content immediate. Camera pose makes use of Plücker embeddings, injected via adaptive layer normalization (AdaLN). Text enters as chunk-wise prompts via cross-attention.

The analysis group claims 4 upgrades over LingBot-World:

• An unbounded interplay horizon with constant output high quality.
• A distilled real-time variant ample to drive 720p video streams at 60 fps.
• A broader motion area, together with attacking, archery, spell-casting, and taking pictures.
• An agentic harness pairing a pilot agent with a director agent.

The major mannequin is 14B. A light-weight 1.3B counterpart is described as deployable on a single GPU.

The Architecture: MoBA and Two-Stage Training

The core contribution is the Mixture of Bidirectional and Autoregressive (MoBA) Attention Mask. It explains the drift.

Standard autoregressive video coaching makes use of a instructor forcing masks. Each noisy body attends to itself and its clear context. The analysis group discovered a failure right here. As context grows, the mannequin leans on that context as a substitute of predicting future frames. The result’s overfitting and visible high quality degradation.

MoBA appends a bidirectional full-attention block to the instructor forcing masks. That block acts as a regularizer. It additionally helps the mannequin deal with flexible-length technology.

The cross-attention masks mirrors the cut up. The autoregressive element attends to a background immediate together with chunk-wise prompts in a lower-triangular sample. That prevents future semantics from leaking backward. The bidirectional element attends to at least one world immediate.

Pre-training optimizes a conditional flow-matching goal with rectified-flow interpolation. Post-training then compresses the multi-step instructor right into a few-step pupil:

1. Consistency distillation: Latents on the identical instructor probability-flow ODE (PF-ODE) trajectory should map to similar predictions.
2. Distribution matching distillation (DMD): The generator follows the KL gradient between noised pupil and noised knowledge distributions.

The essential element sits within the DMD. The analysis group applies it over lengthy self-rollout trajectories, not solely teacher-forced states. The pupil is due to this fact optimized on the state distribution its personal predictions induce. That is the said mechanism behind anti-drift.

The Agentic Harness: The Feature Worth Taking Seriously

A body predictor doesn’t play itself. The Robbyant analysis group wraps the generator in a Director-Pilot Co-Simulation Framework.

As described in the research paper, a Vision-Language Model is the Director. It governs macroscopic semantic guidelines and causal reasoning. The Diffusion Transformer video generator is the Pilot. It simulates low-level bodily dynamics and renders transitions.

The harness exposes two interplay modes:

Mode A: Direct Semantic Interaction. The VLM reads the present body and generates occasion playing cards. No object masks are required.

Mode B: Tracking-Assisted Object Interaction. A SAM-based (Segment Anything Model) action-proposal loop tracks objects throughout chunks. Users choose a tracked object and set off actions. The analysis paper reveals door-opening and ball-rotating rollouts.

Users may also intervene textually. Global state shifts change time of day or climate. Local entity injection spawns creatures, and the VLM picks plausible entry points.

The interface follows sport conventions. WASD drives motion, IJKL controls view. Space triggers a soar; P triggers a wing glide. Keys U and O carry VLM-proposed character actions. Keys F and G carry environmental occasions. Numeric keys are user-registered occasion slots.

Hands-On: What Ships and What Doesn’t

Expectations want calibration right here.

The codebase is built on Wan2.2. Only lingbot-world-v2-14b-causal-fast is downloadable at this time. The causal-pretrained 14B, the bidirectional 14B, and each 1.3B variants are marked TODO.

git clone https://github.com/robbyant/lingbot-world-v2.git
cd lingbot-world-v2
pip set up -r necessities.txt            # torch >= 2.4.0
pip set up flash-attn --no-build-isolation
huggingface-cli obtain robbyant/lingbot-world-v2-14b-causal-fast 
  --local-dir ./lingbot-world-v2-14b-causal-fast

The supplied generate.py runs causal inference with KV caching. It processes frames chunk-by-chunk somewhat than . The reference command is eight-GPU and 480P:

torchrun --nproc_per_node=8 generate.py 
  --task i2v-A14B --size 480*832 
  --ckpt_dir lingbot-world-v2-14b-causal-fast 
  --image examples/03/picture.jpg --action_path examples/03 
  --dit_fsdp --t5_fsdp --ulysses_size 8 
  --frame_num 361 --local_attn_size 18 --sink_size 6 
  --prompt "A serene lakeside scene with a lone tree standing in calm water..."

The launched reference script is 480×832 throughout eight GPUs. The 60 fps determine describes the deployed stream, which passes a spatio-temporal refiner. That refiner upsamples decoded frames, then synthesizes intermediate frames for a better body charge. Both levels compile into TensorRT engines.

A Diffusers checkpoint additionally exists:

import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image, export_to_video

pipe = DiffusionPipeline.from_pretrained(
    "robbyant/lingbot-world-v2-14b-causal-fast",
    dtype=torch.bfloat16, device_map="cuda")
frames = pipe(picture=load_image("seed.png"), immediate="...").frames[0]
export_to_video(frames, "output.mp4")

For a hosted path, Reactor serves the mannequin as reactor/lingbot-world-2. Its docs record 48 fps at 1664×960. Sessions are command-driven and stateful:

from reactor_sdk import Reactor, ReactorStanding

reactor = Reactor(model_name="reactor/lingbot-world-2", api_key=KEY)

@reactor.on_status(ReactorStanding.READY)
async def on_ready(standing):
    ref = await reactor.upload_file("seed.jpg")
    await reactor.send_command("set_image", {"picture": ref})
    await reactor.send_command("set_prompt", {"immediate": "A misty alpine valley."})
    await reactor.send_command("begin", {})

Movement is persistent state, not a pulse. set_move_longitudinal: “ahead” drives till you ship “idle”. Commands land on the subsequent chunk boundary. 

Comparison

The research paper’s comparison is qualitative. Every superiority declare rests on side-by-side body grids.

Use Cases

• Game and degree prototyping: Seed a picture of a canal city. Hot-swap the immediate to summon a snowstorm. Iterate on temper earlier than any asset pipeline exists.
• Embodied simulation: Generate first-person rollouts underneath scripted digital camera poses. Feed frames to a coverage learner with out authoring an Unreal Engine scene.
• Synthetic knowledge for video understanding: Chunk-wise prompts produce temporally localized occasions. Each chunk carries its personal caption by building.
• Agent analysis harnesses: Let the Director agent suggest occasions, then rating how a downstream agent reacts. The paper’s personal analogy is a coding scaffold reminiscent of Codex.
• Previsualization: Drive set_camera_pose for directed agent digital camera strikes. Restyle the world mid-stream with out shedding the reference picture.

Key Takeaways

• LingBot-World-Infinity (LingBot-World 2.0) is Robbyant’s open causal world mannequin, launched with one 14B checkpoint.
• MoBA consideration plus DMD over self-rollout is the actual contribution — it targets long-horizon drift straight.
• A Director (VLM) and Pilot (DiT generator) harness turns the body predictor into an interactive simulator.
• The 720p/60 fps headline wants the unreleased deployment stack; the general public script runs 480×832 on eight GPUs.
• Comparison is qualitative: one 60-minute rollout, no VBench or FVD, and a non-commercial CC BY-NC-SA 4.0 license.

Interactive Explainer