H Company Releases NeoMME: A Family of 260M and 800M Single-Tower Multimodal Encoders That Drop the Vision Tower and Causal Decoder
Most visible doc retrievers in manufacturing at this time are hand-me-downs. ColPali and the fashions that adopted it take a generative vision-language mannequin and repurpose it as an encoder. The consequence nonetheless carries a individually pretrained imaginative and prescient tower and a causal decoder that by no means generates a token. That is parameter and compute overhead for a activity that solely wants representations.
H Company has launched NeoMME, a household of 260M and 800M bidirectional encoders that drops each parts. One Transformer processes multilingual textual content tokens and uncooked 32×32 RGB picture patches via the similar layers, educated from random initialization. The retrieval fine-tune, NeoMME-Retriever, reaches 0.523 nDCG@10 on ViDoRe v3 at 260M parameters.
Is it deployable? Yes. Every checkpoint ships beneath Apache 2.0 with day-zero assist in Hugging Face Transformers. The 260M mannequin indexes 51.3 pages per second on a single NVIDIA L40S and encodes a question in 78.3 ms on a CPU-only host.
One tower, two modalities
Text enters via an ALBERT-style factorized embedding: a 256-dimensional lookup projected to mannequin width. Images are cut up into non-overlapping 32×32 patches and projected by a 2-layer MLP educated from scratch. No patch-merging module, no SigLIP2 tower.
Both fashions assist a 16,384-token context, sufficient for 2 normal 3,840×2,160 4K UHD photographs after patching. Most layers use symmetric sliding-window consideration; each sixth layer and the closing layer attend globally. The stack makes use of grouped-query consideration, query-key normalization, gated consideration, 2D rotary place embeddings, and squared-ReLU MLPs. Exact parameter counts are 262,937,906 and 793,715,032.
The tokenizer is a whitespace-unconstrained BPE with a 131,072-entry vocabulary, educated from scratch. Across 14 goal languages in FLORES-200 devtest, it emits 44.4% fewer tokens than ModernBERT.
Trained as a masked diffusion denoiser
Pretraining is discrete masked diffusion over textual content, optionally conditioned on seen picture patches. Text-only segments draw a corruption fee uniformly from 0 to 1. Multimodal segments draw from 0.30 to 1, which removes the language-only shortcut and forces the mannequin to learn the web page.
A cross-modal ablation probe confirms this works. At 90% masking, seen web page patches increase masked-token accuracy by 38.4 factors for the 260M mannequin and 40.5 factors for the 800M mannequin. Each run processes about 524 billion packed enter tokens, roughly 290 billion of them text-only, on 16 and 32 H100 accelerators respectively.
Retrieval outcomes
NeoMME-Retriever provides two collectively educated heads on the shared spine: a mean-pooled dense head with Matryoshka widths, and a late-interaction head projecting each token and patch to 128 dimensions. One ahead go returns each.
On ViDoRe v3, the 260M mannequin scores 0.523 nDCG@10 and the 800M mannequin 0.556. The 260M consequence sits inside 0.002 of ColQwen2.5-v0.2 at 3.75B parameters, and 26.1 factors above the greatest different sub-300M mannequin. The 800M mannequin lands 0.9 factors behind the equally sized Vultron Retriever Flash. On ViDoRe v1 and v2 the fashions attain 0.860/0.522 and 0.874/0.559 nDCG@5.
Text retrieval is weaker. On BEIR-15, late interplay reaches 0.4881 and 0.5126, towards 0.5722 for LateOn at 149M parameters. The authors attribute this partly to supervision scale: NeoMME noticed roughly 430K textual content question examples, towards roughly 660M contrastive examples for mLateOn.
Storage and throughput
Late-interaction indexes are costly. A 2048×2048 web page yields 4,162 vectors, about 1.5 MB per ViDoRe v3 doc in float32. Two strategies deliver that down. Hierarchical token pooling at issue 10 with int8 queries and paperwork provides 39.0 kB per web page, a 39.4× discount retaining 99.16% of baseline nDCG@10. Pool issue 8 with int8 queries and binary paperwork provides 6.0 kB, a 255.5× discount retaining 95.19%.
Indexing is quick for the vector depend. At a matched 2048×2048 enter on one L40S, NeoMME-260M encodes 51.3 pages per second towards ColModernVBERT’s 26.0, a 1.97× hole.