|

Netflix AI Team Cuts Wide-Partition Read Latency from Seconds to Milliseconds by Splitting Cassandra Partitions Per ID

ByRicardo

Netflix’s engineering crew printed a method for handling wide partitions in Apache Cassandra. The analysis work targets Netflix’s TimeSequence Abstraction, a platform for temporal occasion knowledge.

TL;DR

  • Dynamic partitioning splits vast Cassandra partitions per TimeSequence ID, asynchronously and transparently, with no utility adjustments.
  • Detection runs on the learn path by way of byte counting and a Kafka occasion; splitting targets immutable partitions first.
  • Bloom filters (single-digit microseconds) plus a cached wide_row metadata lookup route reads to the smaller baby partitions.
  • Checksums, retained unique partitions, Data Bridge Spark checks, and shadow comparability guard correctness.
  • Reads improved from seconds to low double-digit milliseconds; tail latency fell to ~200 ms; 500MB+ partitions stayed out there.

What is Dynamic Repartitioning?

Netflix’s TimeSequence Abstraction ingests and queries petabytes of temporal occasion knowledge with millisecond latency. It makes use of Apache Cassandra 4.x because the underlying storage. Cassandra was chosen for throughput, latency, price, and operational maturity.

Time-series knowledge is organized into partitions that group occasions by identifier and time vary. These partitions can develop ‘vast’ as occasions accumulate over time. Dynamic repartitioning splits an outsized partition into smaller baby partitions asynchronously. Applications preserve querying the identical logical partition whereas the storage format evolves transparently.

Why Wide Partitions Hurt Reads

For most datasets, common learn latency stays in single-digit milliseconds. When partitions develop too vast, tail learn latencies rise into seconds. These gradual reads can produce learn timeouts. In excessive instances, clusters see Garbage Collection pauses, excessive CPU utilization, and thread queueing.

TimeSequence servers additionally deal with very excessive learn throughput, which compounds the issue. Scaling up the Cassandra cluster is all the time an possibility. But Netflix crew needed smarter alternate options than simply throwing more cash on the drawback.

The Partitioning Strategy Behind TimeSequence

TimeSequence breaks datasets into discrete time chunks: Time Slices, time buckets, and occasion buckets. This lets Netflix crew question and drop knowledge by time with out creating tombstones. When a namespace (dataset) is created, customers specify anticipated workload traits. A provisioning pipeline runs Monte Carlo simulations to decide infrastructure and partition configuration.

This up-front strategy falls quick in three conditions. Workload might be unknown or inaccurately estimated early in a mission. Workload may evolve as site visitors and product necessities change. Finally, knowledge outliers exist, the place just a few IDs obtain way more occasions.

Discrete Time Slices present a pure escape hatch for the primary two instances. Each new Time Slice can use a unique partitioning technique. But manually tuning 1000’s of datasets is just not sustainable, so automation is required.

Solution 1: Time Slice Re-Partitioning

Cassandra exposes introspection APIs equivalent to nodetool tablehistograms for partition-size percentiles. A background employee displays these histograms and exposes them by a Cassandra digital desk. It computes an adjustment issue when partition sizes miss a configured density. That density is commonly set between 2 MiB and 10 MiB, relying on workload.

For instance, provisioning as soon as chosen 60-second time buckets, producing partitions underneath 10 KB. That over-partitioning triggered excessive learn amplification and thread queueing. The employee updates future Time Slices with a corrected technique:

Copy Code
DynamicTimeSliceConfigWorker:
namespace: my_dataset_1
Observed: TimeSlices have p99 partitions under configured goal of 10MB.
Proposed: time_bucket interval: 60s -> 604800s

This lowered learn latencies and timeouts triggered by thread queueing. But it solely helps when many of the desk warrants re-partitioning. It doesn’t assist when solely a share of IDs are vast. For these partial instances, Netflix crew affords three choices:

  • Do Nothing: the best strategy when top-level metrics present no impression.
  • Partial Returns: aborts an in-flight request breaching a latency SLO, returning knowledge collected to this point.
  • Block IDs: an excessive step for take a look at or spam IDs that destabilize the system.

Block IDs is utilized by configuration, itemizing the offending TimeSequence IDs:

Copy Code
dgwts.config.<dataset>.block.Ids: "<tsid-1>, <tsid-2>, <tsid-3>"

These choices don’t assist when legitimate, necessary IDs develop vast. Those callers nonetheless want each occasion, which is the place Solution 2 applies.

Solution 2: Dynamic Partitioning per ID

Dynamic partitioning is an asynchronous pipeline that splits vast partitions per TimeSequence ID. It operates on the ID stage, not the desk stage. It has three levels: Detection, Planning & Splitting, and Serving Reads.

Detection occurs on the learn path. Every learn tracks bytes learn for a partition. When bytes exceed a configured threshold, the server emits an occasion to Kafka:

Copy Code
{
  "time_slice": "data_20260328",
  "time_series_id": "profileId:123",
  "time_bucket": 7,
  "event_bucket": 2,
  "immutable": true,
  "model": "0"
}

Here, immutable marks a partition not receiving writes. The model subject is reserved for future use, equivalent to invalidation. Netflix crew detects on reads, not writes, as a result of most knowledge by no means wants splitting. Some reads on vast partitions keep gradual for seconds till the pipeline catches up. The preliminary implementation targets immutable partitions to scale back complexity.

Planning reads all the partition as soon as to compute an correct break up plan. Checkpointing lets failed planning reads resume from the final saved level. The wide_row metadata desk shops break up states, checkpoints, and routing info.

Splitting delegates to a technique equivalent to EventBucketPartitionSplitStrategy. It assigns extra occasion buckets to the identical time bucket. For ultra-wide partitions, it caps occasion buckets to management learn amplification. Spreading throughout partitions nonetheless distributes reads over Cassandra replicas.

Validating compares a pre-split checksum in opposition to a post-split checksum. A break up is marked COMPLETED solely when each checksums match. Netflix additionally tracks pre- and post-split partition sizes to affirm splits are sized properly.

The Read Path: Bloom Filters and Metadata Routing

TimeSequence servers periodically load accomplished break up partition-keys into in-memory Bloom filters. Every learn checks the Bloom filter, which responds in single-digit microseconds. That verify is sufficiently small to be virtually invisible to callers. On a success, the server reads wide_row metadata to route the question:

Copy Code
{
  "pre_split_data": {
    "time_slice": "data_20260328",
    "time_series_id": "6313825",
    "time_bucket": 0,
    "event_bucket": 2
  },
  "post_split_data": {
    "time_slice": "wide_data_20260328_0",
    "event_bucket_partition_strategy": {
      "target_event_buckets": 2,
      "start_event_bucket": 32
    }
  }
}

That metadata learn is backed by a read-through cache. The current PartitionReader then serves reads from the smaller break up partitions. Reusing the identical schema minimizes code adjustments, and outcomes are merged earlier than returning. The unique vast partition is rarely deleted, offering a secure fallback for partial failures and eventual consistency.

Netflix crew additionally verified splits offline utilizing Data Bridge Spark jobs. A phased rollout superior by learn modes as confidence grew. Its Comparison section in contrast bytes served by the outdated and new learn paths in shadow mode.

Solution 1 vs Solution 2: Comparison

Dimension Time Slice Re-Partitioning (Solution 1) Dynamic Partitioning per ID (Solution 2)
Granularity Table / Time Slice stage Individual TimeSequence ID stage
Trigger Background employee on partition histograms Bytes learn exceed a threshold on the learn path
Scope of impact Future Time Slices solely Existing immutable vast partitions
Core mechanism Adjust time_bucket interval / goal density Split into baby event-bucket partitions
Detection sign nodetool tablehistograms percentiles Per-read byte counting, Kafka occasion
Best when Most of the desk is mis-partitioned A couple of IDs are vast outliers
Data motion None; applies to new writes Async break up; unique retained as fallback
Target / validation Configured density (2–10 MiB) Pre/post-split checksum + shadow comparability

Use Cases With Examples

  • Long-lived person exercise logs: A profileId:123 accumulating years of playback occasions turns into vast. Solution 2 spreads it throughout occasion buckets, so pagination stays out there. Netflix paginated 500MB+ partitions whereas remaining out there utilizing this path.
  • Device or IoT telemetry: A small set of chatty gadgets dominate occasion quantity. Solution 2 isolates these scorching IDs with out repartitioning the entire desk.
  • Over-provisioned new dataset: A crew guesses 60-second buckets and will get sub-10 KB partitions. Solution 1 widens the interval for future Time Slices, reducing learn amplification.
  • Latency-sensitive dashboards: A caller wants a quick response greater than full knowledge. Partial Returns caps latency on the SLO and returns what was gathered.

Try It: Interactive Splitter Demo

The embedded simulator under fashions the learn path finish to finish. Set a partition dimension, a detection threshold, and a goal event-bucket rely. Running a learn walks by Detection, Planning, Splitting, Validation, and Serving. It reveals the Kafka detection occasion and a modeled latency drop; the numbers are illustrative, not measured.

Run learn</button>
<button class=”btn sec” id=”reset”>Reset</button>
</div>

<div>
<div class=”pipe” id=”pipe”>
<div class=”step” data-i=”0″><span class=”n”>01</span>Detection</div>
<div class=”step” data-i=”1″><span class=”n”>02</span>Planning</div>
<div class=”step” data-i=”2″><span class=”n”>03</span>Splitting</div>
<div class=”step” data-i=”3″><span class=”n”>04</span>Validate</div>
<div class=”step” data-i=”4″><span class=”n”>05</span>Serve reads</div>
</div>

<div class=”stage”>
<p class=”rowlbl”>Original partition (retained as fallback)</p>
<div class=”vast” id=”vast”>profileId:123 — 520 MB</div>
<div class=”kids” id=”children”></div>
<div class=”notice” id=”standing”>Idle. Press <b>Run learn</b> to begin the pipeline.</div>
</div>

<div class=”mrow”>
<div class=”metric gradual”>
<div class=”ok”>Single vast learn</div>
<div class=”v” id=”mSlow”>—</div>
<div class=”s”>one massive partition, sequential</div>
</div>
<div class=”metric quick”>
<div class=”ok”>Split parallel learn</div>
<div class=”v” id=”mFast”>—</div>
<div class=”s”>N smaller partitions, merged</div>
</div>
</div>

<pre id=”json”>// Kafka detection occasion seems right here on a wide-partition hit</pre>
</div>
</div>

<div class=”foot”>
<span>Illustrative simulation of the learn path — latency values are modeled for instructing, not measured manufacturing numbers.</span>
<span><b>Marktechpost</b></span>
</div>
</div>

<script>
(operate(){
var d=doc.getElementById(‘dwp’);
var $=operate(id){return d.querySelector(‘#’+id);};
var sSize=$(‘sSize’),sThr=$(‘sThr’),sBuk=$(‘sBuk’);
var vSize=$(‘vSize’),vThr=$(‘vThr’),vBuk=$(‘vBuk’);
var vast=$(‘vast’),children=$(‘children’),standing=$(‘standing’),json=$(‘json’);
var mSlow=$(‘mSlow’),mFast=$(‘mFast’),run=$(‘run’);
var steps=d.querySelectorAll(‘.step’);
var working=false;

operate sync(){
vSize.textContent=sSize.worth+’ MB’;
vThr.textContent=sThr.worth+’ MB’;
vBuk.textContent=sBuk.worth;
vast.textContent=’profileId:123 — ‘+sSize.worth+’ MB’;
}
[sSize,sThr,sBuk].forEach(operate(s){s.addEventListener(‘enter’,operate(){if(!working){sync();}});});

// illustrative latency mannequin
operate slowMs(mb){ return Math.spherical(30 + mb*mb*0.012); } // grows ~quadratically
operate fastMs(mb,n){ var baby=mb/n; return Math.spherical(8 + baby*1.1 + 0.05); } // dominated by largest baby + bloom(µs)+cache

operate clearPipe(){ steps.forEach(operate(s){s.className=’step’;}); }
operate setStep(i,cls){ steps[i].className=’step ‘+cls; }
operate wait(ms){ return new Promise(operate(r){setTimeout(r,ms);}); }

operate esc(s){return s;}

async operate go(){
if(working) return; working=true; run.disabled=true;
var dimension=+sSize.worth, thr=+sThr.worth, n=+sBuk.worth;
clearPipe(); children.innerHTML=”; vast.className=’vast’;
mSlow.textContent=’—’; mFast.textContent=’—’;

var isWide = dimension > thr;

// 01 Detection
setStep(0,’on’);
standing.innerHTML=’Read path tracks bytes learn for <b>profileId:123</b>…’;
await wait(650);
if(!isWide){
setStep(0,’performed’);
standing.innerHTML=’Bytes learn (<b>’+dimension+’ MB</b>) under threshold (<b>’+thr+’ MB</b>). No break up wanted — served immediately.’;
mSlow.textContent=slowMs(dimension)+’ ms’;
mFast.textContent=slowMs(dimension)+’ ms’;
json.innerHTML=’// Partition underneath threshold — no detection occasion emitted’;
working=false; run.disabled=false; return;
}
json.innerHTML=
‘{n’+
‘ <span class=”ok”>”time_slice”</span>: <span class=”g”>”data_20260328″</span>,n’+
‘ <span class=”ok”>”time_series_id”</span>: <span class=”g”>”profileId:123″</span>,n’+
‘ <span class=”ok”>”time_bucket”</span>: <span class=”b”>7</span>,n’+
‘ <span class=”ok”>”event_bucket”</span>: <span class=”b”>2</span>,n’+
‘ <span class=”ok”>”immutable”</span>: <span class=”b”>true</span>,n’+
‘ <span class=”ok”>”model”</span>: <span class=”g”>”0″</span>n’+
‘}’;
setStep(0,’performed’);
standing.innerHTML=’Bytes learn exceeded threshold → detection occasion emitted to <b>Kafka</b>.’;
await wait(750);

// 02 Planning
setStep(1,’on’);
standing.innerHTML=’Planner reads the complete partition <b>as soon as</b> and checkpoints to <b>wide_row</b> metadata.’;
await wait(850); setStep(1,’performed’);

// 03 Splitting
setStep(2,’on’);
standing.innerHTML=’EventBucketPartitionSplitStrategy assigns <b>’+n+'</b> occasion buckets, protecting complete kind order.’;
var per=(dimension/n);
for(var i=0;i<n;i++){
var el=doc.createElement(‘div’);
el.className=’baby’;
el.innerHTML='<span class=”bar”></span>wide_data_20260328_0 · bucket ‘+(32+i)+’ — ‘+per.toFixed(0)+’ MB’;
children.appendChild(el);
}
var kidEls=children.querySelectorAll(‘.baby’);
for(var j=0;j<kidEls.size;j++){ (operate(ok){setTimeout(operate(){kidEls[k].classList.add(‘present’);},ok*120);})(j); }
await wait(n*120+500); setStep(2,’performed’);

// 04 Validate
setStep(3,’on’);
standing.innerHTML=’Comparing <b>pre-split checksum</b> with <b>post-split checksum</b>…’;
await wait(800);
setStep(3,’performed’);
standing.innerHTML=’Checksums match → break up marked <b>COMPLETED</b>. Partition-keys loaded into Bloom filter.’;
await wait(600);

// 05 Serve reads
setStep(4,’on’);
vast.className=’vast okay’;
standing.innerHTML=’Bloom filter hit (µs) → metadata lookup → <b>PartitionReader</b> reads ‘+n+’ partitions in parallel.’;
for(var b=0;b<kidEls.size;b++){ kidEls[b].querySelector(‘.bar’).type.width=’100%’; }
await wait(700); setStep(4,’performed’);

var sm=slowMs(dimension), fm=fastMs(dimension,n);
animate(mSlow,sm,’ms’,sm>1200);
animate(mFast,fm,’ms’,false);
standing.innerHTML=’Done. Tail learn latency dropped from <b>’+fmt(sm)+'</b> to <b>’+fm+’ ms</b> utilizing ‘+n+’ parallel baby reads.’;
working=false; run.disabled=false;
}

operate fmt(ms){ return ms>=1000 ? (ms/1000).toFixed(1)+’ s’ : ms+’ ms’; }
operate animate(el,goal,unit,massive){
var t0=efficiency.now(), dur=600;
operate tick(now){
var p=Math.min(1,(now-t0)/dur), cur=Math.spherical(goal*p);
el.textContent = (massive && cur>=1000) ? (cur/1000).toFixed(1)+’ s’ : cur+’ ‘+unit;
if(p<1) requestAnimationFrame(tick);
else el.textContent = massive && goal>=1000 ? (goal/1000).toFixed(1)+’ s’ : goal+’ ‘+unit;
}
requestAnimationFrame(tick);
}

run.addEventListener(‘click on’,go);
$(‘reset’).addEventListener(‘click on’,operate(){
if(working)return; clearPipe(); children.innerHTML=”;
vast.className=’vast’; mSlow.textContent=’—’; mFast.textContent=’—’;
json.innerHTML=’// Kafka detection occasion seems right here on a wide-partition hit’;
standing.innerHTML=’Idle. Press <b>Run learn</b> to begin the pipeline.’;
});
sync();
})();
</script>

<script>
(operate(){
operate report(){
var el=doc.getElementById(‘dwp’);
var h=(el?el.offsetHeight:doc.physique.offsetHeight)+40;
guardian.postMessage({mtpDwpHeight:h},’*’);
}
window.addEventListener(‘load’,report);
window.addEventListener(‘resize’,report);
// report after animations/interactions
new MutationObserver(report).observe(doc.getElementById(‘dwp’),{subtree:true,childList:true,attributes:true});
setInterval(report,600);
})();
</script>
“>

Results

Average learn latency for vast partitions dropped from seconds to low double-digit milliseconds. Tail latencies fell from a number of seconds to round 200 ms or higher. Read timeouts dropped, and Netflix crew reported decrease CPU utilization and minimal thread queueing. Overall, the Cassandra clusters grew to become extra secure.

For excessive vast rows, the service may paginate and question 500MB+ partitions whereas remaining out there. Such partitions beforehand triggered fixed timeouts and unavailability blips. A paginated question returns efficiently, buying and selling elevated latency for availability:

Copy Code
{
  "next_page_token": "...",
  "data": [ { "...": "..." } ],
  "response_context": [
    {
      "namespace": "...",
      "time_taken": "41.072410142s"
    }
  ]
}

Netflix crew lists future work, together with splitting mutable vast partitions and reprocessing beforehand failed splits. Two classes stand out. Reducing the floor space of a fancy change and deploying incrementally pays off operationally. Investing in confidence-building mechanisms is justified by the function’s complexity, blast radius, and impression.

Check out the Technical details here. Also, be happy to comply with us on Twitter and don’t neglect to be part of our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to accomplice with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so forth.? Connect with us

The publish Netflix AI Team Cuts Wide-Partition Read Latency from Seconds to Milliseconds by Splitting Cassandra Partitions Per ID appeared first on MarkTechPost.

Similar Posts