Ant Group’s Robbyant Unveils LingBot-VA 2.0: A Causal Video-Action Model Built Natively for Physical AI

Robbyant, the embodied AI unit inside Ant Group, has launched the LingBot-VA 2.0.The first embodied-native basis mannequin. It describes a video-action basis mannequin for generalist robotic manipulation. The analysis group pretrains the entire stack for embodiment as a substitute of fine-tuning a video generator.

What is LingBot-VA 2.0?

Most video-action fashions reuse two elements constructed for digital content material creation. One is a reconstruction-oriented VAE. The different is a bidirectional video-diffusion spine, with an motion module connected.

This creates three limitations. Pixel-reconstruction latents protect look however carry restricted bodily construction. Iterative denoising over video tokens is just too gradual for closed-loop management. Generic video targets by no means train how actions reshape the world.

A fourth mismatch is structural. Backbones use bidirectional consideration, whereas management unfolds strictly ahead in time. LingBot VA Version 1.0 finetuned that stack right into a causal mannequin. Version 2.0 pretrains a causal DiT natively.

Version 1: The Semantic Visual-Action Tokenizer

Building on that motivation, stage one replaces the compression-only VAE. Following RepWAM, the tokenizer provides two targets to reconstruction.

Semantic alignment pulls visible latents towards a frozen Perception Encoder instructor. A latent-action goal extracts compact transition variables between consecutive latents. An inverse dynamics mannequin predicts every latent motion. A ahead dynamics mannequin decodes it right into a transport map plus residual.

World states and actions now share one latent house. Unlabeled net video subsequently carries action-relevant supervision.

Version 2: A Causal DiT With a Sparse MoE Video Stream

On high of that house, model 2 pretrains a causal DiT. It retains the Mixture-of-Transformers format of model 1.0. A video knowledgeable and an motion knowledgeable share one causal self-attention. Each owns a separate feed-forward pathway.

The two streams scale asymmetrically. The video knowledgeable replaces its dense FFN with a sparse MoE routed layer. That layer holds 128 routed SwiGLU specialists, top-8 routing, one shared knowledgeable. Load balancing follows the auxiliary-loss-free Loss-Free Balancing technique. The motion knowledgeable retains a dense FFN at hidden dimension 768.

The video spine is roughly 13.0B parameters, about 1.9B lively. With the motion knowledgeable and MCP heads, coaching covers about 15.3B parameters. Roughly 2.5B activate per token at inference. Training makes use of a rectified-flow goal with a hybrid Muon plus AdamW optimizer.

Where the Training Signal Comes From

Beyond structure, two targets form what the mannequin learns.

Multi-chunk prediction (MCP) fixes myopic supervision. Teacher forcing supervises solely the subsequent chunk, so the mannequin can minimize loss by copying look. MCP attaches three light-weight modules predicting the subsequent three chunks. In ablation it matched the baseline’s 45k-step accuracy in 20k steps, a 2.3x coaching speedup.

Meanwhile, 5 targets are co-trained reasonably than staged: T2I, T2V, TI2VA, ICL, and human-robot co-training. Sampling follows a coarse-to-fine schedule, from look grounding to video-action management. Keeping each goal alive avoids forgetting the sooner priors.

Hierarchical Planning

Chunk-level management can not sequence long-horizon objectives. Above the coverage subsequently sits a VLM planner, LoRA-finetuned with a frozen imaginative and prescient tower. It emits structured JSON: performed, instruction, generation_instruction, local_scene_description. It runs at about 2 Hz behind an asynchronous shared buffer. The coverage reads it at every chunk boundary, so planner latency by no means blocks execution.

Foresight Reasoning

Even with a sparse spine, deployment hits a serial bottleneck. If the robotic waits, mannequin latency turns into management latency.

Foresight Reasoning subsequently runs prediction and execution as asynchronous streams. While the robotic executes chunk a_t, the video knowledgeable imagines its consequence. The motion knowledgeable decodes a_{t+1} from that.

Running forward dangers drift. So every returning commentary is encoded into the true latent z_{t+1}, overwriting the imagined one. A forward-dynamics grounding loss trains the video knowledgeable for this position.

# Pseudocode for the asynchronous rollout (Sec. 2.3.7, Eq. 29).
# Not runnable: coverage, executor and encode() are placeholders.

C = init_kv_cache(encode(obs_0))            # feedback-grounded cache C_t
a = coverage.action_expert(C)                 # chilly begin: first motion chunk a_0

whereas not performed:
    executor.begin(a)                       # execution stream, non-blocking

    C_tmp  = C + [a]                        # prediction stream: C_t u {a_t}
    z_hat  = coverage.video_expert(C_tmp)     # ahead dynamics -> imagined z_{t+1}
    a_next = coverage.action_expert(C_tmp + [z_hat])

    obs = executor.wait()                   # actual commentary of a_t returns
    C   = overwrite(C_tmp, z_hat, encode(obs))   # re-ground: z_hat <- true z_{t+1}
    a   = a_next

Performance

Consequently, analysis covers simulation and actual {hardware}. On RoboTwin 2.0, each mannequin trains on 2,500 clear plus 25,000 randomized demonstrations, throughout 50 duties.

Distillation cuts the video sampler from 5 steps to 2, and the motion sampler from 10 to 2. FP8 TensorRT engines, a paged/ragged KV cache with FlashInfer consideration, and host-side overhead removing provide the remaining.

# Reproduces Table 3 of the report precisely. Runnable as-is.
Okay = 32  # low-level management steps inside one generated chunk

stack = [("BF16 PyTorch async rollout baseline", 927),
         ("+ Consistency distillation",          466),
         ("+ Low-precision compiled execution",  369),
         ("+ Long-horizon attention optimization", 272),
         ("+ Runtime overhead reduction",        142)]

for title, ms in stack:
    print(f"{title:40s} {ms:4d} ms  {spherical(1000 / ms * Okay):4d} Hz")

print("end-to-end speedup:", spherical(927 / 142, 1), "x")

Version 1.0 vs Version 2.0

The tokenizer ablation isolates row one. Swapping the WAN2.2 VAE for the semantic tokenizer lifts a 1.3B mannequin from 78.0 to 86.6.

Use Cases and Examples

Beyond benchmarks, 4 deployment shapes stand out.

• Few-shot onboarding: The report states the mannequin adapts from 10 to fifteen demonstrations. Real-world analysis makes use of 20 teleoperated demos per activity. One multi-task checkpoint covers all 4 evaluated duties.
• Demonstration-conditioned management: In-context studying lets a human demonstration video change the textual content instruction. After finetuning on 4 seen duties, the coverage executed unseen compositions. One instance: “put the calabash into the inexperienced plate.”
• Cheap information scaling: Human-robot co-training retargets hand poses into the robotic motion house. Each hand turns into a digital parallel gripper. The selfish corpus spans 65.4k episodes.
• Reactive management: Demonstrations embody Air Hockey and a conveyor belt, the place the coverage anticipates shifting objects.

Key Takeaways

• Pretrains a causal video-action DiT from scratch as a substitute of adapting a video generator.
• A semantic tokenizer places world states and latent actions in a single aligned house.
• Sparse MoE video stream: ~2.5B of ~15.3B parameters activate per token.
• Foresight Reasoning overlaps prediction with execution, re-grounded on each actual commentary.
• Chunk latency 927 ms to 142 ms; async management 35 Hz to 225 Hz.

Interactive Dynamic Explainer