Meet Needle 2: An Open 45M-Parameter Tool-Calling Model That Ships as a 14MB Binary and Runs a Full Session in 28MB of RAM
Cactus Compute has launched Needle 2, an open 45M-parameter mannequin for instrument calling, gadget use, and structured extraction. The complete mannequin ships as a single 14MB binary that runs a full session in about 28MB of RAM. Weights are educated and deployed at CQ2-bit utilizing Cactus Quants, and the mannequin is sealed inside the corporate’s personal C++ engine, so there is no such thing as a runtime to put in and no obtain at inference time. Reported decode throughput is 500 tokens/sec on a Raspberry Pi 5, 400–1,500 tokens/sec on Meta Quest 3S and Apple Vision Pro, and 300–700 tokens/sec on sub-$200 telephones. The design premise is slender and acknowledged plainly by the workforce: mapping a messy sentence onto a typed operate signature wants no world data and no open-ended prose. That framing is why 45M parameters are sufficient right here, and why the mannequin targets {hardware} with no GPU and no NPU.
Is it deployable?
Yes, Needle 2 ships as prebuilt binaries and a static library for macOS, Linux (x86-64, ARM64, ARMv7, RISC-V, MIPS32el), Windows, Android, iOS/watchOS/tvOS, and WebAssembly. Cactus says Pebble already runs Needle domestically in the Index 01 app for offline voice actions.
- Which corporations: Any workforce delivery firmware or apps on constrained {hardware}. Seed-stage wearable and IoT startups, mid-market consumer-electronics OEMs, robotics groups, and massive gadget makers needing an offline fallback. Cloud-first SaaS groups acquire much less.
- Industries: good dwelling, wearables, low-end cell, automotive in-cabin management, service robotics, retail kiosks and POS, routers and IP cameras, and regulated settings the place audio can not depart the gadget.
- Applications: voice-to-action on screenless gadgets, offline equipment management, receipt and bill area extraction, enum tagging, and native routing that escalates to the cloud solely on low confidence.
Architecture: Simple Attention Network
Needle 2 makes use of what Cactus workforce calls a Simple Attention Network. The recipe replaces the FFN with a Hadamard MLP, retains GQA consideration, provides engram key-value reminiscence from hashed n-gram tables, and makes use of multi-lane hyper-connections. The community is 27 layers and 512 broad. The underlying examine is on arXiv as A Controlled Study of Attention-Only Transformers.
Pretraining used a proprietary 115B-token corpus, with 38B tokens of post-training. The analysis workforce notes LFM2.5-230M was pretrained on 19 trillion tokens.
Needle 2 spends 70 MFLOPs per token, with 35M of 45M parameters matmul-active. LFM2.5 230M spends 460, FunctionGemma 270M spends 540, and Apple FM sits close to 6,000.
Engine, grammar, retrieval, and confidence
Weights by no means decompress into RAM. The 2-bit codes develop inside vector registers and fuse into integer dot merchandise, so the arithmetic path stays int8. One binary probes the CPU at startup and selects a kernel tier: SDOT, NEON, AVX2, RISC-V vectors, wasm SIMD, or scalar.
A byte-level grammar compiled out of your JSON schemas constrains each emitted token. Because the matcher is aware of which tokens are authorized earlier than logits exist, the engine skips as much as 98% of the vocabulary projection on structural tokens.
Attention makes use of a 256-token sliding window, and the system flip plus instrument declarations are pinned as KV sinks. Memory stays close to 28MB regardless of dialog size.
Declare 5 or fewer instruments and they render instantly. Above 5, a contrastive retrieval head embeds every schema as soon as, scores the question per flip, and admits solely the highest 5. Unselected instruments are unreachable, not merely unlikely.
Every response carries a confidence worth, the minimal of a calibrated post-hoc head and the decoding chance of the decision tokens. Off-topic requests return the empty name []. The contract is a threshold: act above it, re-ask or escalate beneath it.
Evaluation
Cactus workforce evaluates on 5 public function-calling benchmarks utilizing ordered strict precise match, the place names, name order, and each argument should match. Needle 2 runs end-to-end by way of the shipped engine at CQ2-bit with retrieval on; baselines run f16 beneath vLLM.
| Benchmark | Needle 2 (CQ2) | LFM2.5 230M | FunctionGemma 270M | Apple FM |
|---|---|---|---|---|
| Mobile Actions (961) | 63.7 | 69.1 | 64.0 | 57.6 |
| DroidCall (200) | 17.0 | 11.0 | 17.5 | — |
| Seal-Tools in-domain (700) | 32.6 | 26.9 | 16.3 | — |
| Seal-Tools OOD (654) | 28.7 | 17.0 | 15.6 | — |
| BFCL v4 single-turn (3,641), total | 42.6 | 60.8 | 46.1 | 61.7 |
Needle 2 leads each Seal-Tools splits and posts 98.3 function-name accuracy on Mobile Actions. It trails on BFCL v4, which Cactus attributes to distribution: its corpus is client gadget actions, not normal or enterprise APIs. Well-formed output fee throughout the three,641 BFCL rows is 93.4. The workforce states two asymmetries upfront: f16 baselines favor them, and job specialization favors Needle.
‘<div class=”n2-bar-t”><div class=”n2-bar-f”></div></div>’+
‘<span class=”n2-bar-v”>’+r[2]+'</span></div>’;
}).be a part of(”);
noteEl.textContent=d.be aware;
var fills=barsEl.querySelectorAll(‘.n2-bar-f’);
d.rows.forEach(operate(r,i){
var w=Math.max(2,(r[1]/max)*100);
if(scale back){fills[i].model.width=w+’%’;}
else setTimeout(operate(){fills[i].model.width=w+’%’;},60*i+40);
});
put up();
}
root.querySelectorAll(‘[data-m]’).forEach(operate(b){
b.addEventListener(‘click on’,operate(){
root.querySelectorAll(‘[data-m]’).forEach(operate(x){x.classList.take away(‘is-on’)});
b.classList.add(‘is-on’); metric=b.dataset.m; drawBars(metric);
});
});
drawBars(‘params’);
/* ———- 02 grammar ———- */
var CALL=[
[‘{“name”:”‘,1],[‘set_lights’,0],[‘”,”arguments”:{“room”:”‘,1],[‘living room’,0],
[‘”,”on”:’,1],[‘true’,0],[‘,”brightness”:’,1],[’30’,0],[‘}}’,1]
];
var time period=doc.getElementById(‘n2-term’), m1=doc.getElementById(‘n2-m1’), m2=doc.getElementById(‘n2-m2′);
var timer=null;
operate resetG(){
if(timer){clearInterval(timer);timer=null;}
time period.innerHTML='<span class=”q”>> dim the lounge to 30</span>’;
m1.textContent=’0%’; m2.textContent=’0′; put up();
}
doc.getElementById(‘n2-reset-g’).addEventListener(‘click on’,resetG);
doc.getElementById(‘n2-decode’).addEventListener(‘click on’,operate(){
if(timer){clearInterval(timer);timer=null;}
var flat=[]; CALL.forEach(operate(seg){
for(var i=0;i<seg[0].size;i++) flat.push([seg[0][i],seg[1]]);
});
time period.innerHTML='<span class=”q”>> dim the lounge to 30</span><br>’;
var out=doc.createElement(‘span’); time period.appendChild(out);
var caret=doc.createElement(‘span’); caret.className=’n2-caret’; caret.textContent=’_’; time period.appendChild(caret);
var i=0, pressured=0;
operate step(){
if(i>=flat.size){ if(timer){clearInterval(timer);timer=null;} caret.take away();
m1.textContent=’2%’; m2.textContent=pressured; put up(); return; }
var ch=flat[i][0], isF=flat[i][1];
var s=doc.createElement(‘span’); s.className=isF?’n2-forced’:’n2-free’;
s.textContent=ch; out.appendChild(s);
if(isF) pressured++;
var pct=Math.spherical(100-(pressured/(i+1))*98);
m1.textContent=pct+’%’; m2.textContent=pressured;
i++;
}
if(scale back){ whereas(i<flat.size) step(); }
else timer=setInterval(step,26);
});
/* ———- 03 retrieval ———- */
var TOOLS=[‘set_lights’,’play_music’,’send_message’,’set_thermostat’,’get_weather’,’set_timer’,
‘call_contact’,’navigate_to’,’take_photo’,’lock_door’,’add_calendar_event’,’control_tv’];
var PICKS=[[0,3,11,1,5],[2,6,10,0,4],[3,1,0,5,4]];
var chipsEl=doc.getElementById(‘n2-chips’);
chipsEl.innerHTML=TOOLS.map(operate(t,i){return ‘<span class=”n2-chip” data-i=”‘+i+'”>’+t+'</span>’}).be a part of(”);
operate retrieve(qi){
var sel=PICKS[qi];
[].slice.name(chipsEl.youngsters).forEach(operate(c,i){
c.classList.take away(‘is-sel’,’is-out’);
var d=sel.indexOf(i)>-1?0:180;
setTimeout(operate(){ c.classList.add(sel.indexOf(i)>-1?’is-sel’:’is-out’); }, scale back?0:d);
});
}
root.querySelectorAll(‘[data-q]’).forEach(operate(b){
b.addEventListener(‘click on’,operate(){
root.querySelectorAll(‘[data-q]’).forEach(operate(x){x.classList.take away(‘is-on’)});
b.classList.add(‘is-on’); retrieve(+b.dataset.q);
});
});
retrieve(0);
/* ———- 04 confidence ———- */
var TURNS=[
[‘dim the living room to 30’,0.94,false],
[‘make it warmer in here’,0.81,false],
[‘put the thing on the other thing’,0.58,false],
[‘text Ada that I am running late’,0.88,false],
[‘what is the capital of Peru?’,null,true]
];
var rowsEl=doc.getElementById(‘n2-rows’), slider=doc.getElementById(‘n2-slider’), television=doc.getElementById(‘n2-tv’);
operate renderRows(){
var th=(+slider.worth)/100; television.textContent=th.toFixed(2);
rowsEl.innerHTML=TURNS.map(operate(t){
if(t[2]) return ‘<div class=”n2-row ref”><span class=”n2-row-q”>’+t[0]+
‘</span><span class=”n2-row-c”>[ ]</span><span class=”n2-verdict v-ref”>empty name</span></div>’;
var okay=t[1]>=th;
return ‘<div class=”n2-row ‘+(okay?’act’:’esc’)+'”><span class=”n2-row-q”>’+t[0]+
‘</span><span class=”n2-row-c”>’+t[1].toFixed(2)+'</span><span class=”n2-verdict ‘+
(okay?’v-act”>execute’:’v-esc”>escalate’)+'</span></div>’;
}).be a part of(”);
put up();
}
slider.addEventListener(‘enter’,renderRows); renderRows();
/* ———- 05 reminiscence ———- */
var turns=1, gv=doc.getElementById(‘n2-gv’), gf=doc.getElementById(‘n2-gf’);
var win=doc.getElementById(‘n2-win’), develop=doc.getElementById(‘n2-grow’);
var memv=doc.getElementById(‘n2-memv’), growv=doc.getElementById(‘n2-growv’);
var ofEl=doc.getElementById(‘n2-of’), turnsEl=doc.getElementById(‘n2-turns’);
operate renderMem(){
turnsEl.textContent=’turns: ‘+turns;
var w=Math.min(24, 12+turns*2.5);
win.model.width=w+’%’;
memv.textContent=’28.0 MB’;
gf.model.width=’78%’; gv.innerHTML='<b>28.0 MB</b> / exhausting ceiling 28 MB’;
var g=Math.min(100, 18+turns*7);
develop.model.width=g+’%’;
growv.textContent=(Math.spherical((18+turns*7)*0.62*10)/10)+’ MB’;
ofEl.textContent = g>=78 ? ‘Unbounded KV has crossed the 28MB ceiling. Needle 2 has not moved.’ : ”;
put up();
}
doc.getElementById(‘n2-turn’).addEventListener(‘click on’,operate(){turns++;renderMem();});
doc.getElementById(‘n2-reset-m’).addEventListener(‘click on’,operate(){turns=1;renderMem();});
renderMem();
})();
</script>
</physique>
</html>
“>
Key Takeaways
- Needle 2 is a 45M-parameter tool-calling mannequin in a 14MB binary, operating a session in ~28MB RAM.
- CQ2-bit is educated in from pretraining, not utilized post-hoc, so the deployed mannequin is the educated mannequin.
- It leads Seal-Tools in-domain (32.6) and out-of-domain (28.7), and trails on BFCL v4 total (42.6).
- A 256-token sliding window with pinned instrument sinks makes RAM a mounted ceiling, not a rising curve.
- Confidence scores and empty-call refusals flip edge-to-cloud escalation into an specific product resolution.
Check out the GitHub Repo, Hugging Face Model Card, Cactus Needle page and Paper. Also, be happy to observe us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so on.? Connect with us
The put up Meet Needle 2: An Open 45M-Parameter Tool-Calling Model That Ships as a 14MB Binary and Runs a Full Session in 28MB of RAM appeared first on MarkTechPost.
