|

Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed

ByRicardo
▶

Retrieval high quality in an AI search product is bounded by two issues: how good the embedding mannequin is, and how cheaply you may run it throughout an index. This week, Perplexity Engineering group printed Fast Embeddings on GPUs, an under-the-hood account of the second — the serving infrastructure behind pplx-embed and the rating fashions used throughout Perplexity Search, Computer and the API Platform.

Perplexity group states that embedding inference on the GPU facet has largely converged throughout engines on mature Hopper and Blackwell {hardware}. The wins sit within the runtime and harness across the mannequin: CUDA graph administration, an async result-tracking abstraction, and a Rust request path.

Two site visitors patterns, one engine

Perplexity frames embedding serving as two workloads. Batch embedding occurs when constructing or re-indexing the vector database, the place throughput minimizes value. Online embedding occurs at question time, the place a brief question have to be embedded quick. Scoring sits in between: after vector search, giant doc batches are ranked, balancing each.

The key resolution is that Perplexity didn’t construct a separate embedding engine. Because embedding fashions are small Transformers, batch embedding resembles compute-bound prefill and on-line embedding, usually a couple of tokens, resembles memory-bound decode. So the analysis group reuses the prefill and decode kernels from its LLM stack.

Ivy, Tulip and ROSE

Three companies deal with a request:

  • Ivy is a Rust HTTP gateway. It does the CPU-side work — JSON parsing, tokenization, enter templating, batch splitting — and interprets requests right into a customized gRPC protocol. It additionally splits large-batch requests into chunks and load-balances them throughout replicas, which corrects the load imbalance that arises when manufacturing payloads differ in dimension.
  • Tulip is the inference server interface: a gRPC server constructed with Rust, tokio and tonic, dealing with scheduling and batching earlier than dispatching to the engine.
  • ROSE (Runtime-Optimized Serving Engine) implements mannequin inference. It is primarily Python, gives kernels, layers and mannequin definitions, manages CUDA graphs, and exposes a step() perform to Tulip.

Why the scheduler is intentionally easy

Tulip picks sequences first-come, first-served whereas requests accumulate. That simplicity is justified by a measurement: for small embedding fashions on the sequence lengths Perplexity serves, the linear value of dense layers dominates the quadratic value of consideration. Latency is subsequently roughly proportional to token rely, not sequence rely. Once a batch saturates the GPU, round 512 tokens on a sub-billion-parameter mannequin, packing in additional sequences doesn’t enhance effectivity.

CUDA graphs and LazyTensors

On small batches, CPU-side kernel launching can outweigh GPU execution. Perplexity builds whole-model CUDA graphs for all embedding fashions, capturing each launch right into a single driver name. Because embedding fashions are small, the inflection level the place GPU work exceeds launch value arrives at batches of 1000’s of tokens and tens of sequences. Some consideration implementations block full-model graphs by relying on dynamic host-side inputs; Perplexity upstreamed changes to FlashInfer to allow seize.

Graphs have to be captured per configuration, so token counts are padded to buckets which might be multiples of 64 or 256. That nonetheless yields 1000’s of graphs and a number of minutes of seize per mannequin. The repair is lazy seize: every configuration will get an keen warmup run, then triggers seize and replay on its second hit. This prices p99 latency at startup however spreads minutes of keen work throughout hours.

The second piece is the LazyTensor, which tracks a page-locked host buffer plus a cudaMemcpyAsync and a CUDA occasion. Instead of step() blocking on the gadget, it returns a LazyTensor, letting a Rust async activity wait on batch N whereas the CPU enqueues N+1.

Send a request</button></div>
</div>
</div>

<div class=”pe-panel” id=”peP1″>
<div class=”pe-card”>
<div class=”pe-txt”>On small batches, CPU-side kernel launches can outweigh GPU work. A complete-model <b>CUDA graph</b> captures each launch into one name to the driving force, so the CPU is freed to enqueue the following batch. Toggle the 2 modes.</div>
<div class=”pe-ctl” fashion=”margin:0 0 12px”>
<button class=”pe-btn ghost on” id=”peEager”>Eager launches</button>
<button class=”pe-btn ghost” id=”peGraph”>CUDA graph</button>
</div>
<div class=”pe-lane”><div class=”pe-lbl”>Host / CPU</div><div class=”pe-track” id=”peCpuT”></div></div>
<div class=”pe-lane”><div class=”pe-lbl”>Device / GPU</div><div class=”pe-track” id=”peGpuT”></div></div>
<div class=”pe-stats”>
<div class=”pe-stat”><div class=”v” id=”peLaunches”>—</div><div class=”okay”>Driver calls</div></div>
<div class=”pe-stat”><div class=”v” id=”peGap”>—</div><div class=”okay”>GPU idle gaps</div></div>
</div>
<div class=”pe-txt” fashion=”margin:12px 0 0;font-size:11.5px;shade:#6E8285″>Schematic. Block widths illustrate the launch-overhead sample described within the publish, not measured timings.</div>
</div>
</div>

<div class=”pe-panel” id=”peP2″>
<div class=”pe-card”>
<div class=”pe-txt”>Reading outcomes again usually forces a bunch sync. A <b>LazyTensor</b> tracks a page-locked host buffer plus an async device-to-host copy and a CUDA occasion, so Tulip can block on batch N whereas the CPU already prepares batch N+1.</div>
<div class=”pe-lane”><div class=”pe-lbl”>CPU — put together / sync</div><div class=”pe-track” id=”peLzC”></div></div>
<div class=”pe-lane”><div class=”pe-lbl”>GPU — ahead go</div><div class=”pe-track” id=”peLzG”></div></div>
<div class=”pe-ctl”>
<button class=”pe-btn” id=”peLzRun”>▶ Run 3 batches</button>
<button class=”pe-btn ghost on” id=”peLzOn”>Overlapped</button>
<button class=”pe-btn ghost” id=”peLzOff”>Blocking</button>
</div>
<div class=”pe-note” fashion=”margin-top:12px” id=”peLzNote”>Overlapped: whereas the GPU chews batch N, the CPU is already tokenizing and packing batch N+1.</div>
</div>
</div>

<div class=”pe-panel” id=”peP3″>
<div class=”pe-card”>
<div class=”pe-txt”>For small embedding fashions at these sequence lengths, the linear value of dense layers dominates the quadratic value of consideration, so latency tracks <b>token rely, not sequence rely</b>. Past roughly <b>512 tokens</b> on a sub-1B mannequin, the GPU is saturated and packing in additional sequences stops serving to.</div>
<div class=”pe-lbl” fashion=”margin-top:6px”>Tokens in batch: <span id=”peTokV” fashion=”shade:#3FB6C4″>512</span></div>
<enter kind=”vary” id=”peTok” min=”32″ max=”4096″ step=”32″ worth=”512″>
<div class=”pe-lbl”>GPU utilisation</div>
<div class=”pe-bar”><div class=”pe-fill” id=”peUtil”></div></div>
<div class=”pe-stats”>
<div class=”pe-stat”><div class=”v” id=”peUtilV”>—</div><div class=”okay”>Saturation</div></div>
<div class=”pe-stat”><div class=”v” id=”peState”>—</div><div class=”okay”>Regime</div></div>
</div>
<div class=”pe-txt” fashion=”margin:12px 0 0;font-size:11.5px;shade:#6E8285″>Illustrative curve. The ~512-token saturation level is the determine said within the publish; the form between factors is a stand-in, not a benchmark.</div>
</div>
</div>

<div class=”pe-foot”>
<span>Source: Perplexity Engineering, “Fast Embeddings on GPUs” (Sep 4, 2026)</span>
<span><a href=”https://www.marktechpost.com”>Built by Marktechpost</a></span>
</div>

<script>
(perform(){
var R=doc.getElementById(‘pplxEmbedExplainer’);
var NOTES=[
‘<b>Ivy</b> — parses JSON, tokenizes with the in-house unigram tokenizer, applies input templating and splits large batches, then translates to a custom gRPC protocol. It also load-balances chunks across replicas.’,
‘<b>Tulip</b> — Rust gRPC server on tokio and tonic. Requests accumulate while it dispatches or waits; sequences are picked first-come, first-served and packed into a batch for the accelerator.’,
‘<b>ROSE</b> — the Runtime-Optimized Serving Engine. Python-defined kernels and layers, CUDA-graph management, and a step() function that returns a handle to the GPU computation. No KV cache is allocated for embeddings.’
];
perform q(s){return R.querySelector(s)} perform qa(s){return R.querySelectorAll(s)}
/* tabs */
qa(‘.pe-tab’).forEach(perform(t){t.addEventListener(‘click on’,perform(){
qa(‘.pe-tab’).forEach(perform(x){x.classList.take away(‘on’)});t.classList.add(‘on’);
qa(‘.pe-panel’).forEach(perform(p,i){p.classList.toggle(‘on’,i==+t.dataset.p)});});});
/* 1 movement */
var observe=q(‘#peNote’),dot=q(‘#peDot’);
perform sel(i){qa(‘.pe-node’).forEach(perform(n,okay){n.classList.toggle(‘sizzling’,okay==i)});observe.innerHTML=NOTES[i];}
qa(‘.pe-node’).forEach(perform(n){n.addEventListener(‘click on’,perform(){sel(+n.dataset.i)})});
sel(0);
var busy=false;
q(‘#peGo’).addEventListener(‘click on’,perform(){
if(busy)return;busy=true;var steps=[[0,’3%’],[1,’40%’],[2,’76%’]],okay=0;
dot.fashion.transition=’none’;dot.fashion.left=’3%’;dot.fashion.opacity=’1′;
sel(0);
var iv=setInterval(perform(){okay++;if(okay>2){clearInterval(iv);dot.fashion.opacity=’0′;busy=false;return;}
dot.fashion.transition=’left .8s cubic-bezier(.4,0,.2,1)’;dot.fashion.left=steps[k][1];sel(okay);},900);
});
/* 2 cuda graphs */
var cpuT=q(‘#peCpuT’),gpuT=q(‘#peGpuT’),mode=’keen’;
perform blk(p,l,w,cls,txt)
perform drawG(){
cpuT.innerHTML=”;gpuT.innerHTML=”;
if(mode==’keen’){
for(var i=0;i<6;i++){blk(cpuT,1+i*16.4,7,’pe-cpu’,’launch’);blk(gpuT,8.4+i*16.4,7.6,’pe-gpu’,’kernel’);if(i<5)blk(gpuT,16+i*16.4,7.2,’pe-idle’,’idle’);}
q(‘#peLaunches’).textContent=’6′;q(‘#peGap’).textContent=’5′;
}else{
blk(cpuT,1,12,’pe-cpu’,’graph launch’);blk(cpuT,15,26,’pe-cpu’,’put together subsequent batch’);
blk(gpuT,13.5,84,’pe-gpu’,’6 kernels — one replay, no gaps’);
q(‘#peLaunches’).textContent=’1′;q(‘#peGap’).textContent=’0′;
}}
q(‘#peEager’).addEventListener(‘click on’,perform(){mode=’keen’;q(‘#peEager’).classList.add(‘on’);q(‘#peGraph’).classList.take away(‘on’);drawG();});
q(‘#peGraph’).addEventListener(‘click on’,perform(){mode=’graph’;q(‘#peGraph’).classList.add(‘on’);q(‘#peEager’).classList.take away(‘on’);drawG();});
drawG();
/* 3 lazytensor */
var lzC=q(‘#peLzC’),lzG=q(‘#peLzG’),lzMode=’on’,lzBusy=false,lzNote=q(‘#peLzNote’);
perform drawLz(){
lzC.innerHTML=”;lzG.innerHTML=”;
for(var i=0;i<3;i++){
if(lzMode==’on’){blk(lzC,2+i*32,14,’pe-cpu’,’prep ‘+(i+1));blk(lzG,17+i*32,26,’pe-gpu’,’batch ‘+(i+1));}
else{blk(lzC,2+i*32,12,’pe-cpu’,’prep ‘+(i+1));blk(lzC,15+i*32,16,’pe-idle’,’wait’);blk(lzG,15+i*32,15,’pe-gpu’,’batch ‘+(i+1));}
}}
perform lzRun(){
if(lzBusy)return;lzBusy=true;drawLz();
var bs=lzG.querySelectorAll(‘.pe-blk’),cs=lzC.querySelectorAll(‘.pe-blk’);
[].forEach.name(bs,perform(b){b.fashion.opacity=’.15′});[].forEach.name(cs,perform(b){b.fashion.opacity=’.15′});
var all=[].concat([].slice.name(cs),[].slice.name(bs)),j=0;
var iv=setInterval(perform(){if(j>=all.size){clearInterval(iv);lzBusy=false;return;}all[j].fashion.opacity=’1′;j++;},220);
}
q(‘#peLzRun’).addEventListener(‘click on’,lzRun);
q(‘#peLzOn’).addEventListener(‘click on’,perform(){lzMode=’on’;q(‘#peLzOn’).classList.add(‘on’);q(‘#peLzOff’).classList.take away(‘on’);lzNote.innerHTML=’Overlapped: whereas the GPU chews batch N, the CPU is already tokenizing and packing batch N+1.’;drawLz();});
q(‘#peLzOff’).addEventListener(‘click on’,perform(){lzMode=’off’;q(‘#peLzOff’).classList.add(‘on’);q(‘#peLzOn’).classList.take away(‘on’);lzNote.innerHTML=’Blocking: each step() waits for the gadget, so the CPU sits idle and the GPU begins late.’;drawLz();});
drawLz();
/* 4 batch form */
var tok=q(‘#peTok’);
perform drawT(){
var v=+tok.worth;q(‘#peTokV’).textContent=v;
var u=Math.min(100,Math.spherical(100*(1-Math.exp(-v/230))));
q(‘#peUtil’).fashion.width=u+’%’;q(‘#peUtilV’).textContent=u+’%’;
q(‘#peState’).textContent=v<512?’Under-filled’:(v<1200?’Saturated’:’Throughput-bound’);
q(‘#peState’).fashion.shade=v<512?’#C7A24E’:’#3FB6C4′;
}
tok.addEventListener(‘enter’,drawT);drawT();
})();
</script>
</div>

<script>
(perform(){perform h(){var e=doc.getElementById(‘pplxEmbedExplainer’);if(!e)return;father or mother.postMessage({pplxEmbedH:e.offsetHeight+40},’*’);}
window.addEventListener(‘load’,h);setTimeout(h,300);setTimeout(h,1200);
doc.addEventListener(‘click on’,perform(){setTimeout(h,250)});
doc.addEventListener(‘enter’,perform(){setTimeout(h,120)});
if(window.ResizeObserver){var e=doc.getElementById(‘pplxEmbedExplainer’);if(e)new ResizeObserver(h).observe(e);}})();
</script>
</physique></html>”>

Kernels nonetheless matter

ROSE helps a number of consideration backends for ragged inputs: FlashInfer 2, FlashInfer 3 and FlashAttention 4. Perplexity group reviews FlashAttention 4 is usually quicker, however FlashInfer 3 outperforms it on Qwen-based fashions at very lengthy sequence lengths, so backend choice is made case by case. Notably, when serving an embedding mannequin ROSE doesn’t instantiate a KV cache and dispatches to ragged consideration variants to keep away from padding.

Benchmarks

Perplexity benchmarks in opposition to vLLM v0.22.0 in BF16 on actual weights and eval-derived inputs, with warmup runs verifying cosine similarity divergence inside 0.1%. Four suites are charted: low-latency embeddings (batch 1; 128/512/4096 tokens), low-latency scoring (batch 5/25/50 at 512 tokens), high-throughput embeddings (batch 100, 4 concurrent processes) and high-concurrency embeddings (1 to 16 concurrent requests, together with Ivy tokenization and community overhead).

Key Takeaways

  • Perplexity’s embedding stack reuses its LLM prefill/decode kernels slightly than operating a separate engine.
  • Latency tracks token rely, not sequence rely; ~512 tokens saturates a sub-1B mannequin.
  • Whole-model CUDA graphs plus lazy seize minimize launch overhead with out minutes-long startup.
  • LazyTensor overlaps CPU batch prep with in-flight GPU work as a substitute of blocking on sync.
  • Ivy, Tulip and ROSE are inner; pplx-embed is reachable by way of Perplexity’s Embeddings API.

Check out the Technical details. Also, be happy to observe us on Twitter and don’t neglect to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to accomplice with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so on.? Connect with us

The publish Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed appeared first on MarkTechPost.

Similar Posts