Supabase Releases Evals: an Open Source Benchmark That Scores Claude Code, Codex and OpenCode on Real Supabase Tasks
Supabase has open sourced Supabase Evals, its benchmark and framework for testing how nicely AI brokers construct utilizing Supabase. It runs coding brokers together with Claude Code, Codex, and OpenCode in opposition to actual duties, comparable to constructing a schema, debugging a failed Edge Function, or fixing a damaged RLS coverage, then scores the end result. It powers the general public leaderboard at supabase.com/evals and an inner regression suite monitored day by day.
Is it deployable?
Yes, at the moment. supabase/evals is public below Apache-2.0 and runs regionally through pnpm.
- Industries: Developer tooling, cloud infrastructure, information platforms, and regulated backends in fintech or healthcare, the place an agent writing a incorrect RLS coverage is a safety incident.
- Applications: Regression-testing docs and talent edits, gating SDK releases, and evaluating agent harnesses face to face.
- Constraints: Local-stack runs want a Docker daemon, supplier API keys, and ports 54321–54329 free.
How the harness works
Supabase outlined three dimensions: merchandise (database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api), matters (RLS, safety, migrations, SQL, SDK, observability, self-hosting, exams, declarative-schema), and phases (construct, deploy, examine, resolve). It then picked the smallest situation set touching every dimension as soon as, grounded in assist tickets, bug studies, and GitHub points.
Scenarios cut up into two suites. Benchmark eventualities cowl breadth and are printed. Regression eventualities cowl recognized failure modes, refresh day by day, and don’t transfer printed scores.
Every situation runs in opposition to an actual atmosphere. The framework boots a hosted-like stack and a neighborhood CLI mission in containers, so brokers name the precise MCP server and CLI. A platform-lite runtime exposes a Management API-compatible floor backed by @supabase/lite. Scoring combines deterministic checks with LLM-as-a-judge. Agents get one retry earlier than grading.
Each eval listing holds PROMPT.md (process plus frontmatter), EVAL.ts (the scorer), and elective distant/ and native/ beginning states. Shipping a native/ workspace, or declaring interface: cli, boots a Docker sandbox with the actual CLI put in.
d.onclick=perform(){choose(i)};movement.appendChild(d);
});
perform choose(i){
all(‘#movement .node’).forEach(perform(n,j){n.classList.toggle(‘on’,j===i)});
$(‘#det’).innerHTML='<h4>’+STAGES[i].h+'</h4><p>’+STAGES[i].t+'</p>’;
}
choose(0);
var busy=false;
$(‘#run’).onclick=perform(){
if(busy)return;busy=true;this.disabled=true;
var nodes=all(‘#movement .node’);nodes.forEach(perform(n){n.classList.take away(‘lit’)});
var i=0;var t=setInterval(perform(){
if(i>0)nodes[i-1].classList.take away(‘lit’);
if(i>=nodes.size){clearInterval(t);busy=false;$(‘#run’).disabled=false;return}
nodes[i].classList.add(‘lit’);choose(i);i++;
},900);
};
/* anatomy */
var tree=$(‘#tree’);
FILES.forEach(perform(f,i){
var d=doc.createElement(‘div’);d.className=’f’;
d.innerHTML='<div class=”fn”>’+f.n+'</div><div class=”fd”>’+f.d+'</div>’;
d.onclick=perform(){
all(‘#tree .f’).forEach(perform(x,j){x.classList.toggle(‘on’,j===i)});
$(‘#det2′).innerHTML='<h4>’+f.h+'</h4><p>’+f.t+'</p>’;
};
tree.appendChild(d);
});
all(‘#tree .f’)[0].click on();
/* runtimes */
perform drawRun(okay){
var L=$(‘#lanes’);L.innerHTML=”;
RUN[k].lanes.forEach(perform(x,i){
var d=doc.createElement(‘div’);d.className=’lane’;
d.innerHTML='<span class=”lb”>’+x[0]+'</span><span class=”lt”>’+x[1]+'</span>’;
L.appendChild(d);setTimeout(perform(){d.classList.add(‘act’)},120*i+60);
});
$(‘#rnote’).innerHTML=RUN[k].be aware;
}
all(‘[data-r]’).forEach(perform(b){b.onclick=perform(){
all(‘[data-r]’).forEach(perform(x){x.classList.take away(‘on’)});
b.classList.add(‘on’);drawRun(+b.dataset.r);
}});
drawRun(0);
/* findings */
var rows=$(‘#rows’);
SCORES.forEach(perform(s){
var d=doc.createElement(‘div’);d.className=’row’;
d.innerHTML='<div class=”rn”>’+s.n+'</div><div class=”monitor”><div class=”bar”></div></div><div class=”rv”>—</div>’;
rows.appendChild(d);
});
perform drawScores(on){
all(‘#rows .row’).forEach(perform(r,i){
var v=on?SCORES[i].on:SCORES[i].off;
r.querySelector(‘.bar’).fashion.width=v+’%’;
r.querySelector(‘.rv’).textContent=v+’%’;
});
}
all(‘[data-s]’).forEach(perform(b){b.onclick=perform(){
all(‘[data-s]’).forEach(perform(x){x.classList.take away(‘on’)});
b.classList.add(‘on’);drawScores(b.dataset.s===’1′);
}});
drawScores(true);
/* auto-resize */
perform ping(){
attempt{dad or mum.postMessage({mtpEmbed:’supabase-evals’,top:doc.physique.offsetHeight+40},’*’)}catch(e){}
}
window.addEventListener(‘load’,ping);
setInterval(ping,400);
doc.addEventListener(‘click on’,perform(){setTimeout(ping,60)});
</script>
“>
Findings
Agents go most eventualities with no talent loaded. In the Build stage, Opus 5 and Kimi K3 each scored 100% unaided. Skills closed the remainder of the hole: Sonnet 5 rose from 78% to 100%, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89%.
Three weaknesses surfaced. Agents hand-write migrations as an alternative of utilizing declarative schemas, prompting a skill guidance update. Agents confirm auth by hand moderately than reaching for @supabase/server, prompting a package selection guide. And docs utilization varies sharply: Codex / GPT-5.6 reads roughly 8 docs pages per situation versus about 2 for Claude Code, which checks docs in below 40% of eventualities even with expertise loaded.
Key Takeaways
- Supabase open sourced
supabase/evalsbelow Apache-2.0. - Scenarios run in opposition to actual containerized Supabase stacks, not mocks.
- Scoring mixes deterministic checks with LLM-as-a-judge; one retry allowed.
- Skills mattered least for prime fashions, most for smaller ones.
Check out the Technical details and GitHub Repo. Also, be at liberty to comply with us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us
The submit Supabase Releases Evals: an Open Source Benchmark That Scores Claude Code, Codex and OpenCode on Real Supabase Tasks appeared first on MarkTechPost.
