|

Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour

ByRicardo

How do you benchmark an online search API when the factor being examined can learn the reply key? A search agent has a fetch device. If the gold labels sit in a public dataset, the agent can obtain them mid-evaluation and skip retrieval solely. A comparable drawback arises when the solutions are already encoded within the mannequin’s parametric reminiscence: an accurate response not demonstrates that net search labored. Keenable’s reply is NEEDLE, a stay open-source benchmark that rebuilds its question set from recent public sources quite than freezing one. News queries are regenerated hourly from RSS feeds and Google Trends; finance, scholar, authorized, and rare-entity queries are regenerated day by day from SEC XBRL, arXiv, Europe PMC, CourtListener, and public agent logs. Fifteen search APIs run in opposition to the identical question textual content beneath one protocol, and each rating is learn in opposition to final, a pooled oracle engine that marks what the entire subject managed to search out.

Is it reproducible?

Yes, as an open supply analysis harness quite than a product. needle is a Python CLI put in with uv sync and pushed by two subcommands per benchmark, generate and run. It wants an OpenRouter key for judging and one API key per engine examined, and runs on a laptop computer or in CI. It permits recreated all question streams which are getting used along with the rating high quality judgements.

What NEEDLE measures

NEEDLE stands for News, Everyday, Expert, Deep-tail, and Legal Evaluation. Each vertical fashions a special agent intent. News tasks the most recent merchandise from ~124 curated RSS feeds and Google Trends right into a key phrase question. Finance asks registry info from Wikidata and GLEIF plus single-quarter 10-Q figures from SEC XBRL. Scholar turns one paper into 4 question types: a degraded title, a full-text-only element, a natural-language clue, and a hedged tip-of-the-tongue description. Deep-tail samples rare-word queries from public agent-trajectory releases together with DeepResearchGym, OpenResearcher and LRAT. Legal pulls current CourtListener opinions throughout 14 federal courts and eCFR sections.

Scoring splits alongside the identical line. News and deep-tail haven’t any single right outcome, so an LLM decide charges every outcome 0 to 4 and the harness studies nDCG@5 with a duplicate-URL penalty. Finance studies answer-recall@5: does the very fact attain the agent inside a top-5 snippet. Scholar and authorized are known-item duties scored by identifier match.

The ceiling is the fascinating half

Every engine receives the identical question textual content. The runner points one name at a time, so latency percentiles are comparable and no engine takes concurrent load. Judging occurs on the engine’s personal rating, titles and snippets. Pages are by no means fetched and outcomes are by no means re-ranked. Evidence is clipped to 2,000 characters for everybody, and the decide doesn’t see the engine identify.

The extra fascinating quantity is the final ceiling. For every question, NEEDLE swimming pools the outcomes returned by each engine into an artificial oracle engine, then orders that mixed set by relevance. That creates an empirical ceiling based mostly on what your complete subject was capable of retrieve.

The hole to final is subsequently an higher certain on agentic search high quality because it stands at present. A giant hole means higher outcomes existed however each engine didn’t floor or rank them properly. A weak final rating means one thing totally different: even after pooling each supplier, the benchmark discovered little sturdy proof. In different phrases, NEEDLE can distinguish a rating drawback from a retrieval drawback shared by the entire market.

Where the sphere truly stands

Numbers beneath are printed 7-day means for the window ending 2026-08-28.

Finance is near solved: Exa 0.910, Keenable 0.872, Perplexity 0.871, Google 0.847, in opposition to an final of 0.965. Scholar spreads out, Keenable 0.774 to Tavily 0.310 in opposition to a 0.869 ceiling, as a result of title queries are answerable from metadata and physique queries aren’t. Deep-tail is hardest and closest to actual agent visitors: Exa leads at 0.557 of final, Keenable follows at 0.470, Bing sits at 0.199. The hole between delivered and achievable high quality widens as queries method how brokers truly search.

Latency is one other essential metric right here, as a result of brokers name search dozens of instances per job. Same window: Keenable-realtime 193 ms p50 / 284 ms p95, Exa 1,876 / 2,955, Bing 2,767 / 9,381.

Key Takeaways

  • NEEDLE regenerates queries hourly for information and day by day for the opposite 4 verticals, so there is no such thing as a mounted set to overfit.
  • Five verticals, 15 search APIs, one protocol: identical question textual content, identical 2,000-character proof cap, one request at a time.
  • Every leaderboard is learn in opposition to final, a pooled oracle engine marking the ceiling the entire subject reached.
  • On rare-entity queries from actual agent logs the highest engine reaches 0.557 of that ceiling; on finance most engines cluster between 0.77 and 0.91.
  • Code is MIT, runs execute in public GitHub Actions, and per-run artifacts ship to a Hugging Face dataset.

Check out the live dashboard, the GitHub repo, the technical write-up, and the archived artifacts. All credit score goes to the researchers of this challenge.

Also, be happy to comply with us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to accomplice with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The submit Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour appeared first on MarkTechPost.

Similar Posts