Ant Group’s Robbyant Open-Sources LingBot-Vision: A 1B Boundary-Centric Vision Foundation Model for Dense Spatial Perception

Robbyant, the embodied-AI firm inside Ant Group, has open-sourced LingBot-Vision, a household of self-supervised Vision Transformers constructed for dense spatial notion. The weights ship beneath Apache-2.0 on Hugging Face in 4 sizes — ViT-giant, ViT-large, ViT-base, and ViT-small — along with a technical report and inference code.

Most imaginative and prescient basis fashions are skilled for semantic invariance: they be taught to reply what is in a picture whereas discarding precisely the fine-grained spatial construction — object boundaries, contours, depth discontinuities — that robots and different bodily embodied techniques depend upon. LingBot-Vision inverts that precedence. It treats boundaries as a native pretraining sign relatively than a downstream output, and the payoff is a 1B-parameter spine that matches or surpasses fashions as much as 7× bigger on dense spatial duties, together with the 7B DINOv3.

What is LingBot-Vision?

LingBot-Vision is a self-supervised pretrained encoder for spatially structured downstream duties. The flagship ViT-g/16 has roughly 1.1B parameters and is skilled with a brand new goal referred to as masked boundary modeling on a curated corpus of about 161M photos — chosen from a 2B internet pool — with no human labels, no exterior edge detectors, and no pretrained spine to bootstrap from. The coaching can be notably economical: the corpus is an order of magnitude smaller than DINOv3’s LVD-1689M, and the mannequin consumes lower than a 3rd of DINOv3’s coaching samples.

The encoder outputs dense patch-token options supposed for frozen readouts. For deployment at smaller budgets, the flagship is distilled into ViT-L (300M), ViT-B (86M), and ViT-S college students that lead dense prediction inside their dimension courses.

How Masked Boundary Modeling Works

The methodology builds on the DINO/iBOT self-distillation paradigm: a trainer — an EMA copy of the scholar — generates on-line targets, and the scholar recovers them from masked views.

Standard masked picture modeling hides patches at random, ignoring what every patch depicts. A flat inside patch is reasonable to get better from its neighbors; a patch straddling an object boundary carries construction that context alone can not provide. Boundaries are the least redundant, most informative areas of a picture — and random masking treats them like all the pieces else.

LingBot-Vision closes that hole with two concepts.

Boundary-forcing. The trainer predicts a dense boundary discipline on-line and identifies the boundary-bearing tokens B. These are compelled into the scholar’s masked set on prime of the random masks M, giving the mixed masks M⁺ = M ∪ B. Masked tokens are then routed by geometry: boundary tokens obtain an express geometric goal along with the semantic self-distillation goal, whereas inside masked tokens maintain the usual semantic goal alone. This routing issues as a result of a semantic goal is inherently ambiguous precisely the place two areas meet — the geometric goal is well-posed exactly the place standard masked modeling is weakest, which is what lets semantic and geometric representations co-emerge relatively than compete.

Categorical boundary discipline. Boundaries are modeled as line segments lifted right into a dense discipline: each close by pixel shops an attribute vector a(p) = (d, θ, φ¹, φ²) recording its distance to the closest section and three angles that find it. Directly regressing this discipline in a trainer–pupil loop collapses. The repair is to discretize every channel into Okay = 32 bins, recasting boundary prediction as per-pixel classification — which lets the boundary department inherit the identical centering and sharpening equipment that stabilizes trendy self-distillation.

The categorical kind has a sublime aspect impact. Under the classical a-contrario null speculation of “no construction,” boundary orientations are uniformly distributed — and that null is now actually the uniform distribution over bins. Deviation from uniformity is proof of an actual boundary, so a parameter-free Number-of-False-Alarms (NFA) take a look at validates each decoded section at no additional value. The trainer exploits this at every iteration: it decodes candidate segments from its personal discipline prediction, retains solely the NFA-validated survivors, and re-renders them into the goal discipline — so unsupported construction by no means turns into a educating sign.

The full goal sums 4 phrases:

L = L_DINO + λᵢ · L_iBOT + λᵦ · L_bnd + λₖ · L_KoLeo

Benchmarks and Performance

All dense outcomes beneath use frozen options with a single linear layer, so efficiency is attributable to the illustration relatively than a decoder.

On NYU-Depth v2, LingBot-Vision posts the most effective RMSE of the complete comparability (0.296), forward of the 7B DINOv3 (0.309) with roughly 7× fewer parameters, and forward of the 2B V-JEPA 2.1 (0.307). On KITTI it’s the finest mannequin beneath 2B parameters. On semantic segmentation it’s on par with the distilled DINOv3 ViT-H+ — 1.3 mIoU behind on ADE20K, matching on Cityscapes, forward on VOC12 — whereas enhancing over the same-size DINOv2 by 4+ mIoU on all three benchmarks; the one remaining hole is to the DINOv3 household itself (2.4 mIoU on ADE20K to the 7B mannequin), whose dense power comes from distillation and devoted dense-feature targets.

Video object segmentation makes use of training-free label propagation over frozen options. LingBot-Vision reaches 70.0 J&F on DAVIS-2017 and 73.5 on YouTube-VOS — on par with DINOv3 ViT-H+ (71.1 / 74.0) and the 7B DINOv3 (71.1 / 74.1), and the most effective amongst all remaining fashions at any scale. The boundary tokens themselves are steady sufficient to be tracked by video by plain cosine similarity of frozen options, with no temporal supervision.

The trade-off is image-level recognition: ImageNet-1K linear probing reaches 86.32 and k-NN 83.39, trailing DINOv3-7B, which spends its capability on image-level invariance. The benefits additionally survive distillation — the 0.3B ViT-L pupil matches the 7B DINOv3 on NYUv2 depth (0.310 vs. 0.309) with about 23× fewer parameters.

Use Cases and How to Load It

The frozen patch tokens serve a number of dense workloads immediately: depth estimation reads geometry straight from the options, semantic segmentation advantages from function transitions that land precisely on object contours, and video object segmentation works by cosine-similarity token matching. The encoder additionally serves because the initialization for downstream depth-completion coaching.

Loading a spine follows the official repository:

git clone https://github.com/robbyant/lingbot-vision.git
cd lingbot-vision
conda create -n lingbot-vision python=3.10 -y
conda activate lingbot-vision
python -m pip set up -r necessities.txt
python -m pip set up -e .

import torch
from lingbot_vision import load_pretrained_backbone, extract_patch_tokens, load_image


machine = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if machine == "cuda" else torch.float32


# Downloads mannequin.pt from Hugging Face on first use.
spine, embed_dim = load_pretrained_backbone(
    variant="small",   # big | giant | base | small; defaults to giant
    machine=machine,
    dtype=dtype,
)


img_norm, _, _ = load_image(
    "examples/instance.png",
    dimension=512,
    patch_size=spine.patch_size,
    mode="sq.",
)
patch_tokens, patch_grid = extract_patch_tokens(spine, img_norm, machine, dtype)


print(patch_tokens.form, patch_grid, embed_dim)
# torch.Size([1, 1024, 384]) (32, 32) 384

The variant argument selects the dimensions and defaults to giant. Output patch_tokens has form [B, H*W, C]. Requirements are Python ≥ 3.10 and PyTorch ≥ 2.0, with a GPU really helpful for the bigger backbones.

LingBot-Depth 2.0: The Downstream Payoff

To present what a spatial-perception-native encoder buys downstream, the crew upgraded its depth-completion system to LingBot-Depth 2.0. The masked-depth-modeling recipe is unchanged from model 1.0; precisely two components moved: the encoder initialization switched from DINOv2 to LingBot-Vision (in ViT-L and ViT-g variants), and the curated coaching knowledge grew from the publicly launched 3M samples to 150M.

Those two modifications set main outcomes throughout 14 depth-completion benchmarks spanning block-mask, sparse, and real-sensor regimes. On block-masked DIODE-Indoor, RMSE is halved from 0.132 to 0.062. The system is strongest on the transparent-object ClearGrasp captures (0.010 / 0.012 RMSE) — the basic failure case of energetic depth sensing.

Notably, the 2 modifications compound relatively than cancel: as coaching knowledge grows from 3M to 150M, the DINOv2-initialized curve saturates past 20M samples whereas the LingBot-Vision curve retains enhancing. More knowledge amplifies, relatively than washes out, the benefit of the higher place to begin.

Key Takeaways

• LingBot-Vision makes boundaries a local pretraining sign, bootstrapped from uncooked photos with no labels, edge detectors, or pretrained backbones.
• Boundary-forcing plus a categorical boundary discipline lets geometry and semantics co-emerge — and yields a parameter-free NFA validation take a look at for free.
• The 1B spine posts the most effective NYU-Depth v2 RMSE in its comparability, forward of the 7B DINOv3, whereas coaching on an order-of-magnitude smaller corpus.
• The benefits survive distillation: the 0.3B ViT-L matches the 7B DINOv3 on NYUv2 with ~23× fewer parameters.
• Swapping solely the encoder and scaling knowledge took LingBot-Depth 2.0 to main outcomes on 14 depth-completion benchmarks, and the encoder’s edge widens with extra knowledge.
• Weights ship beneath Apache-2.0 in ViT-g/L/B/S sizes for each deployment finances.

Interactive Dynamic Explainer

Check out the Paper and Model Weights. Also, be at liberty to observe us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to accomplice with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so forth.? Connect with us

_{Note:Thanks to the Ant Research crew for the thought management/ Resources for this text. Ant Research crew has supported this content material/article for promotion.}

The submit Ant Group’s Robbyant Open-Sources LingBot-Vision: A 1B Boundary-Centric Vision Foundation Model for Dense Spatial Perception appeared first on MarkTechPost.