|

Meta Superintelligence Labs Introduces REFRAG: Scaling RAG with 16× Longer Contexts and 31× Faster Decoding

ByRicardo

Table of contents

A staff of researchers from Meta Superintelligence Labs, National University of Singapore and Rice University has unveiled REFRAG (REpresentation For RAG), a decoding framework that rethinks retrieval-augmented era (RAG) effectivity. REFRAG extends LLM context home windows by 16× and achieves as much as a 30.85× acceleration in time-to-first-token (TTFT) with out compromising accuracy.

Why is lengthy context such a bottleneck for LLMs?

The consideration mechanism in giant language fashions scales quadratically with enter size. If a doc is twice as lengthy, the compute and reminiscence price can develop fourfold. This not solely slows inference but additionally will increase the dimensions of the key-value (KV) cache, making large-context functions impractical in manufacturing techniques. In RAG settings, most retrieved passages contribute little to the ultimate reply, however the mannequin nonetheless pays the complete quadratic worth to course of them.

How does REFRAG compress and shorten context?

REFRAG introduces a light-weight encoder that splits retrieved passages into fixed-size chunks (e.g., 16 tokens) and compresses every right into a dense chunk embedding. Instead of feeding hundreds of uncooked tokens, the decoder processes this shorter sequence of embeddings. The result’s a 16× discount in sequence size, with no change to the LLM structure.

https://arxiv.org/pdf/2509.01092

How is acceleration achieved?

By shortening the decoder’s enter sequence, REFRAG reduces the quadratic consideration computation and shrinks the KV cache. Empirical outcomes present 16.53× TTFT acceleration at okay=16 and 30.85× acceleration at okay=32, far surpassing prior state-of-the-art CEPE (which achieved solely 2–8×). Throughput additionally improves by as much as 6.78× in comparison with LLaMA baselines.

How does REFRAG protect accuracy?

A reinforcement studying (RL) coverage supervises compression. It identifies probably the most information-dense chunks and permits them to bypass compression, feeding uncooked tokens straight into the decoder. This selective technique ensures that vital particulars—similar to actual numbers or uncommon entities—aren’t misplaced. Across a number of benchmarks, REFRAG maintained or improved perplexity in comparison with CEPE whereas working at far decrease latency.

What do the experiments reveal?

REFRAG was pretrained on 20B tokens from the SlimPajama corpus (Books + arXiv) and examined on long-context datasets together with Book, Arxiv, PG19, and ProofPile. On RAG benchmarks, multi-turn dialog duties, and long-document summarization, REFRAG persistently outperformed sturdy baselines:

  • 16× context extension past normal LLaMA-2 (4k tokens).
  • ~9.3% perplexity enchancment over CEPE throughout 4 datasets.
  • Better accuracy in weak retriever settings, the place irrelevant passages dominate, as a result of skill to course of extra passages beneath the identical latency finances.
https://arxiv.org/pdf/2509.01092

Summary

REFRAG exhibits that long-context LLMs don’t must be gradual or memory-hungry. By compressing retrieved passages into compact embeddings, selectively increasing solely the necessary ones, and rethinking how RAG decoding works, Meta Superintelligence Labs has made it attainable to course of a lot bigger inputs whereas working dramatically quicker. This makes large-context functions—like analyzing complete experiences, dealing with multi-turn conversations, or scaling enterprise RAG techniques—not solely possible however environment friendly, with out compromising accuracy.

FAQs

Q1. What is REFRAG?
REFRAG (REpresentation For RAG) is a decoding framework from Meta Superintelligence Labs that compresses retrieved passages into embeddings, enabling quicker and longer-context inference in LLMs.

Q2. How a lot quicker is REFRAG in comparison with current strategies?
REFRAG delivers as much as 30.85× quicker time-to-first-token (TTFT) and 6.78× throughput enchancment in comparison with LLaMA baselines, whereas outperforming CEPE.

Q3. Does compression scale back accuracy?
No. A reinforcement studying coverage ensures vital chunks stay uncompressed, preserving key particulars. Across benchmarks, REFRAG maintained or improved accuracy relative to prior strategies.

This fall. Where will the code be accessible?
Meta Superintelligence Labs will launch REFRAG on GitHub at facebookresearch/refrag

Check out the PAPER here. Feel free to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Also, be at liberty to observe us on Twitter and don’t overlook to affix our 100k+ ML SubReddit and Subscribe to our Newsletter.

The submit Meta Superintelligence Labs Introduces REFRAG: Scaling RAG with 16× Longer Contexts and 31× Faster Decoding appeared first on MarkTechPost.

Similar Posts