|

Liquid AI Releases LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: Bidirectional Encoders That Stay Fast at 8K Context on CPU

ByRicardo

Liquid AI has launched two open-weight bidirectional encoders, LFM2.5-Encoder-230M and LFM2.5-Encoder-350M. Both are masked language fashions constructed on the LFM2 hybrid spine. Both carry an 8,192-token context.

Encoders sit beneath classifiers, intent routers, security filters, and PII detectors. Those jobs run repeatedly, normally with out a GPU, and more and more on longer inputs. BERT established the category. ModernBERT pushed its accuracy, pace, and context. Liquid AI’s argument is that the LFM2 structure continues that line, as a result of its value grows extra slowly as inputs get longer.

How a decoder turned an encoder

The encoders will not be skilled from scratch. They are initialized from the LFM2.5-230M and LFM2.5-350M decoder backbones, then transformed with three adjustments:

  • First, the causal consideration masks is changed with a bidirectional one, so each token attends to either side.
  • Second, the LFM2 quick convolutions are made non-causal utilizing symmetric middle padding, so every token’s convolution mixes in neighbours on either side.
  • Third, the mannequin is skilled with a masked language modeling goal at a 30% masks price. That is denser than BERT’s 15%, following proof Liquid AI cites {that a} larger masks price helps at this scale.

Training runs in two phases:

  • Stage one establishes common language competence with a short-context MLM goal on a big internet corpus at 1,024 tokens.
  • Stage two extends context to eight,192 tokens on the total knowledge combine, strengthening factual, authorized, and multilingual competence.

Architecturally, the spine interleaves gated short-convolution blocks with grouped-query consideration, the identical design described within the LFM2 technical report. Both checkpoints use a hidden dimension of 1024 and a 65,536-token vocabulary, and assist 15 languages. The license is the LFM Open License v1.0.

The embed under breaks down the conversion, the total rating desk, the per-task scores, and the CPU latency figures.


The benchmark outcomes

Liquid AI evaluated 14 fashions on 17 duties pulled from GLUE, SuperGLUE, and multilingual classification. Every mannequin is absolutely fine-tuned per process, and the reported rating is that fine-tuned mannequin’s consequence.

LFM2.5-Encoder-350M posts a 17-task imply of 81.02 (±1.00), rating fourth. The three fashions forward of it are all bigger. They are XLM-R XL at 3.5B (83.06), ModernBERT-large at 395M (81.68), and XLM-R massive at 560M (81.34). The prime mannequin is sort of 10 occasions its dimension.

LFM2.5-Encoder-230M posts 79.29 (±1.02), rating sixth. It beats ModernBERT-base at 78.19 and each EuroBERT mannequin within the desk, together with EuroBERT-610M (75.87) and EuroBERT-2.1B (72.19). Both new encoders additionally rating above Liquid AI’s personal retrieval siblings, LFM2.5-ColBERT-350M (76.18) and LFM2.5-Embedding-350M (75.68). That hole is the acknowledged cause Liquid AI constructed a general-purpose encoder as an alternative of reusing the retrievers.

The methodology is the extra attention-grabbing half, and it’s open-sourced under Apache-2.0. Every mannequin is loaded with fp32 grasp weights and bf16 autocast, so the desk compares fashions relatively than quantity codecs. Every mannequin makes use of the identical AdamW recipe, taken from the EuroBERT card. Learning price is chosen per mannequin and process throughout 10 charges and 3 seeds. Scores are then reported because the imply over 5 contemporary seeds that by no means touched choice. The transformers model is pinned to 4.56.2 so dependency drift will not be an uncontrolled variable.

Use Cases and Deployment Environments

The launch names three settings. Edge and embedded units come first. A automobile’s onboard compute or an industrial controller has no spare GPU, and can’t afford a cloud spherical journey. Regulated and on-premise techniques in finance, healthcare, and authorized, the place paperwork are lengthy, delicate, and can’t go away in-house infrastructure. And high-volume cost-sensitive pipelines, the place a small encoder acts as an affordable first cross in entrance of a bigger mannequin.

Liquid AI additionally places a helpful quantity on the context window: 8,192 tokens is roughly 13 to fifteen pages. One ahead cross covers a full contract or a whole affected person document.

To present what a fine-tuned encoder appears to be like like, the analysis workforce shipped 5 demos. Each runs in a CPU-only Hugging Face Space. They cowl zero-shot prompt routing, zero-shot policy linting, and spell checking. A PII detector handles 40 info varieties throughout 16 languages. A bonus masked-diffusion demo runs the encoder as a chatbot that generates by iteratively unmasking.

Getting them operating

Both encoders load by way of transformers. The physique is uncovered as Lfm2BidirectionalModel and masked-LM loading makes use of Lfm2BidirectionalForMaskedLM. Both are wired by way of auto_map, so trust_remote_code=True is required on each load name.

A base encoder produces general-purpose representations, not process outputs, so fine-tuning is obligatory. Liquid AI’s fine-tuning tutorial walks by way of lengthy authorized paperwork at an 8k context configuration. The mannequin choice steering is easy: 350M when accuracy issues most, 230M for tighter {hardware} or larger throughput.


Key Takeaways

  • Liquid AI launched LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, open-weight bidirectional encoders with 8,192-token context.
  • Both convert LFM2.5 decoders into encoders by way of bidirectional masking, non-causal quick convolutions, and 30% MLM.
  • LFM2.5-Encoder-350M ranks 4th of 14 fashions at 81.02; the three above it are all bigger.
  • At 8K tokens on CPU, the 230M runs one ahead cross in ~28s versus over 90s for ModernBERT-base.

Check out the Liquid AI blog post, the Hugging Face write-up, and the mannequin playing cards for LFM2.5-Encoder-230M and LFM2.5-Encoder-350M.

The submit Liquid AI Releases LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: Bidirectional Encoders That Stay Fast at 8K Context on CPU appeared first on MarkTechPost.

Similar Posts