Moonshot AI and Tsinghua Researchers Propose PrfaaS: A Cross-Datacenter KVCache Architecture that Rethinks How LLMs are Served at Scale
For years, the way in which massive language fashions deal with inference has been caught inside a field — actually. The high-bandwidth RDMA networks that make trendy LLM serving work have confined each prefill and decode to the identical datacenter, typically even the identical rack. A staff of researchers at Moonshot AI and Tsinghua University is making the case that this constraint is about to interrupt down — and that the best structure can already exploit that shift.
The analysis staff introduces Prefill-as-a-Service (PrfaaS), a cross-datacenter serving structure that selectively offloads long-context prefill to standalone, compute-dense prefill clusters and transfers the ensuing KVCache over commodity Ethernet to native PD clusters for decode. The outcome, in a case research utilizing an inner 1T-parameter hybrid mannequin, is 54% greater serving throughput than a homogeneous PD baseline and 32% greater than a naive heterogeneous setup — whereas consuming solely a fraction of obtainable cross-datacenter bandwidth. The analysis staff be aware that in comparison at equal {hardware} value, the throughput achieve is roughly 15%, reflecting that the complete 54% benefit comes partly from pairing higher-compute H200 GPUs for prefill with H20 GPUs for decode.
Why the Existing Architecture Has Hit a Wall
To perceive what PrfaaS solves, it helps to grasp why LLM serving is break up into two phases within the first place. Prefill is the step the place the mannequin processes all the enter tokens and generates the KVCache — it’s compute-intensive. Decode is the place the mannequin generates output tokens one at a time — it’s memory-bandwidth-intensive. Prefill-decode (PD) disaggregation separates these two phases onto totally different {hardware}, which improves utilization and permits every part to be independently optimized.
The drawback is that separating prefill from decode creates a transport drawback. Once prefill runs on one set of machines and decode runs on one other, the KVCache produced by prefill should be transferred to the decode facet earlier than output era can start. In standard dense-attention fashions — these utilizing Grouped Query Attention (GQA) — this KVCache is gigantic. The analysis staff benchmarks MiniMax-M2.5, a consultant dense mannequin with GQA, producing KVCache at roughly 60 Gbps for a 32K-token request on a single 8×H200 occasion. That quantity of knowledge requires RDMA-class interconnects to switch with out stalling compute, which is why standard PD disaggregation is tightly sure to a single datacenter-scale community cloth. Moving prefill and decode to separate clusters, not to mention throughout datacenters, has merely not been possible.
Hybrid Attention Changes the Math
What makes PrfaaS well timed is an architectural shift occurring at the mannequin degree. A rising class of fashions — together with Kimi Linear, MiMo-V2-Flash, Qwen3.5-397B, and Ring-2.5-1T — undertake hybrid consideration stacks that interleave a small variety of full-attention layers with a bigger variety of linear-complexity or bounded-state layers similar to Kimi Delta Attention (KDA), Multi-head Latent Attention (MLA), and Sliding Window Attention (SWA). In these architectures, solely the full-attention layers produce KVCache that scales with sequence size. The linear-complexity layers preserve fixed-size recurrent states whose footprint is negligible at lengthy context.
The KV throughput numbers — outlined as KVCache dimension divided by prefill latency — inform the story clearly. At 32K tokens, MiMo-V2-Flash produces KVCache at 4.66 Gbps versus 59.93 Gbps for MiniMax-M2.5, a 13× discount. Qwen3.5-397B reaches 8.25 Gbps versus 33.35 Gbps for Qwen3-235B, a 4× discount. For Ring-2.5-1T particularly, the paper decomposes the financial savings: MLA contributes roughly a 4.5× compression over GQA, and the 7:1 hybrid ratio contributes one other roughly 8× discount, yielding an total KV reminiscence saving of roughly 36×. For the inner 1T mannequin used within the case research, KV throughput at 32K tokens is simply 3.19 Gbps — a degree that trendy inter-datacenter Ethernet hyperlinks can really maintain.
But the analysis staff is cautious to make a distinction that issues for AI devs constructing actual programs: a smaller KVCache is critical however not ample to make cross-datacenter PD disaggregation sensible. Real workloads are bursty, request lengths are skewed, prefix caches are distributed inconsistently throughout nodes, and inter-cluster bandwidth fluctuates. A naive design that routes each prefill to a distant cluster nonetheless runs into congestion and unstable queuing.
What PrfaaS Actually Does
The PrfaaS-PD structure sits on prime of three subsystems: compute, community, and storage. The compute subsystem separates clusters into two varieties — native PD clusters that deal with end-to-end inference for brief requests, and PrfaaS clusters with high-compute-throughput accelerators devoted to long-context prefill. The community subsystem makes use of intra-cluster RDMA for quick native transfers and commodity Ethernet for cross-cluster KVCache transport. The storage subsystem builds a distributed hybrid prefix cache pool that handles linear consideration recurrent states (request-level, fixed-size, exact-match solely) and full-attention KVCache blocks (block-level, rising linearly with enter size, supporting partial prefix matching) in separate teams backed by a unified block pool.
The key routing mechanism is length-based threshold routing. Let l denote the incremental prefill size of a request after subtracting any cached prefix, and t a routing threshold. If l > t, the request goes to the PrfaaS cluster and its KVCache is shipped over Ethernet to a decode node. If l ≤ t, it stays on the native PD path. In the case research, the optimum threshold is t = 19.4K tokens, which routes roughly 50% of all requests — the longer ones — to the PrfaaS cluster.
Making the Ethernet path dependable in follow requires extra than simply low KV throughput. The analysis staff specifies three concrete transport mechanisms: layer-wise prefill pipelining to overlap KVCache era with transmission, multi-connection TCP transport to completely make the most of out there bandwidth, and congestion monitoring built-in with the scheduler to detect loss and retransmission indicators early and forestall congestion accumulation.
On prime of this, the analysis staff introduces a dual-timescale scheduler. At brief timescales, it displays PrfaaS egress utilization and queue depth, adjusting routing when the hyperlink approaches its bandwidth ceiling. It additionally handles cache-affine routing: when bandwidth is scarce, every cluster’s prefix cache is evaluated independently; when bandwidth is ample, the scheduler considers one of the best cached prefix throughout all clusters and performs a cross-cluster cache switch if it reduces redundant computation. At longer timescales, the scheduler rebalances prefill and decode node counts throughout the native PD cluster as site visitors patterns shift, protecting the system close to the throughput-optimal working level.
The Numbers
In the case research, a PrfaaS cluster of 32 H200 GPUs is paired with an area PD cluster of 64 H20 GPUs, related by a VPC community offering roughly 100 Gbps of cross-cluster bandwidth. The combination PrfaaS egress load below the optimum configuration is roughly 13 Gbps — simply 13% of obtainable Ethernet capability — and the paper notes that the PrfaaS cluster stays compute-bound with substantial bandwidth headroom to spare. The analysis additionally initiatives this to bigger deployments: even at the size of a ten,000-GPU datacenter, the mixture egress bandwidth required for KVCache switch totals solely about 1.8 Tbps, properly throughout the capability of recent inter-datacenter hyperlinks.
Mean Time to First Token (TTFT) drops by 50% and P90 TTFT drops by 64% in comparison with the homogeneous baseline. The naive heterogeneous configuration — all prefill on H200, all decode on H20, with no routing or scheduling logic — achieves just one.16× throughput over the homogeneous baseline, in comparison with 1.54× for the complete PrfaaS-PD system. The hole between 1.16× and 1.54× isolates the contribution of the scheduling layer and reveals it accounts for almost all of the sensible achieve.
The analysis staff positions PrfaaS not as a near-future idea however as a design that is viable as we speak for hybrid-architecture fashions — and argues that as context home windows develop, KVCache compression methods mature, and phase-specialized {hardware} similar to NVIDIA’s Rubin CPX for prefill and LPU-style chips for decode grow to be extra extensively out there, the case for cross-datacenter PD disaggregation will solely strengthen.
Check out the Paper here. Also, be at liberty to comply with us on Twitter and don’t overlook to affix our 130k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us
The submit Moonshot AI and Tsinghua Researchers Propose PrfaaS: A Cross-Datacenter KVCache Architecture that Rethinks How LLMs are Served at Scale appeared first on MarkTechPost.
