Perplexity AI Releases WANDR: An Open Benchmark Evaluating Research Agents That Must Search Wide And Deep

Research brokers already deal with actual data work at this time. Teams delegate aggressive mapping, due diligence, and literature assessment to them. However, most benchmarks check a single reply, not giant evidence-backed collections. Perplexity targets that hole with a brand new open benchmark.

Perplexity launched WANDR (Wide ANd Deep Research). It is an open benchmark and analysis harness. It is constructed round 500 lifelike, difficult data-collection duties for data work. WANDR is the extensive sibling of Perplexity’s DRACO benchmark for deep analysis. DRACO asks whether or not an agent produces an correct, full, goal long-form report. WANDR as an alternative asks whether or not it may possibly construct a big assortment with proof.

What is WANDR

At its core, WANDR checks two calls for collectively. Wide means discovering a big, usually open-ended set of qualifying entities. Deep means investigating each entity sufficient to assist every declare with proof. Combining each adjustments the issue for brokers. Just a few compelling examples are usually not sufficient right here. A cultured narrative constructed on incomplete analysis additionally falls brief.

To seize this, WANDR makes use of a composable qualification key hierarchy. One process would possibly request firm(n) -> worker(m) -> url(ok). This means n qualifying corporations, m staff every, and ok supporting pages every. Every full path by means of the tree will get validated independently. The similar construction can symbolize a flat listing, nested search, or matrix.

A Concrete Task Example

To floor that hierarchy, contemplate the launched ceo_cfo_appointments process. It asks for not less than 70 US-based corporations. Each will need to have a CEO or CFO appointment first introduced between March 1 and April 30, 2026. For every, the agent provides one authoritative appointment web page. A subtask provides a listing-authority web page per firm. Together, the duty requires 140 source-backed information.

Concretely, the 2 hierarchies and one submitted file appear like this:

# Task hierarchies
firm(70) -> company_appointee(1) -> url(1)   # 70 appointment information
firm(70) -> url(1)                           # 70 itemizing information

# One file the grader re-fetches and re-checks (values are illustrative)
{
  "merchandise":     "Example Corp - new CFO",
  "url":      "https://issuer.instance.com/press/cfo-appointment",
  "excerpts": ["Example Corp today named Jane Doe as Chief Financial Officer, effective April 2026."],
  "reply":   "Jane Doe appointed CFO; introduced April 2026"
}

Realistic Tasks, Generated At Scale

Beyond single examples, WANDR builds its duties from actual utilization. It begins from de-identified patterns seen in manufacturing, not artificial prompts. A semi-automated pipeline then turns these patterns into duties. The pipeline runs 4 phases: seeding, authoring, admission, and curation. It makes use of an interleaved author-critic loop with mechanical linting.

As a outcome, the median process asks for 50 members and 245 information total. Across all 500 duties, WANDR requires 170,495 source-backed information. Tasks break up into 167 decrease, 166 center, and 167 larger problem. Difficulty relies on per-record work, not scale alone.

How WANDR Grades Agents

Unlike fastened reply keys, WANDR grades every declare towards cited proof. Every file accommodates an merchandise, URL, chosen excerpts, and reply. The grader re-fetches the web page throughout analysis. It checks whether or not the web page is usable and in scope. It then verifies the excerpts really seem and assist each requirement.

These binary file verdicts then roll up by means of the hierarchy. Precision measures the standard of what a system submitted. Recall measures quality-adjusted completion, filling any shortfall with zeros. Soft scores give partial credit score to incomplete members. Hard scores rely solely members whose full subtree is appropriate.

The Benchmark Results

Using that methodology, Perplexity ran six manufacturing programs on all 500 duties. Its personal Search as Code (SaC) system leads. Still, no system comes near fixing the benchmark.

With extra effort, Perplexity reaches 0.447 comfortable F1 on the xhigh setting. Cost throughout settings spans greater than 4 orders of magnitude. It ranges from $0.03 per process as much as $324.83 per process.

Beyond the leaderboard, 4 findings stand out. First, partial progress is widespread, however full protection isn’t. Every system reveals comfortable recall under comfortable precision. Second, scale compounds the issue sharply. Deeper hierarchies damage most, since every department provides a failure level. Third, discovery is the primary structural bottleneck. Top-level discovery completion ranges from 0.611 to 0.951 throughout programs. Under-delivery, not duplicate merging, explains most lacking quantity. Fourth, discovering a usable web page is often straightforward. Turning it into full proof is the arduous half. For Perplexity, 41.4% of pages miss a substantive requirement. Also, 57.5% of excerpts fail to assist the total declare. Its comfortable F1 falls from 0.531 underneath a retrieval-only verify to 0.363 underneath the total verdict.

Notably, Search as Code matches this process form nicely. An agent can specific retrieval, filtering, fan-out, joins, deduplication, and stopping logic as a program. Deterministic compute then handles repeated operations outdoors the mannequin context.

Use Cases With Examples

Practically, WANDR maps to jobs groups already automate. A market analyst wants each qualifying competitor, with matching proof for every. A due-diligence group wants dozens of corporations, then possession, executives, and financing. Talent sourcing wants many candidates, every with supporting profile pages. WANDR checks precisely these wide-and-deep assortment patterns at skilled scale.

Because grading is per-record, groups can localize failures exactly. The rating tree isolates loss to discovery, enrichment, or proof extraction. This analysis helps engineers enhance one weak stage at a time.

Key Takeaways

• WANDR is an open benchmark with 500 evidence-heavy, wide-and-deep duties.
• Tasks use a qualification key hierarchy validated path by path.
• Grading is reference-free; the grader re-fetches and checks cited proof.
• Perplexity Search as Code leads at 0.363 comfortable F1 and 0.133 arduous F1.
• Discovery and full proof stay the largest failure factors.

Check out the Technical details and Repo here. Also, be happy to comply with us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so forth.? Connect with us

The submit Perplexity AI Releases WANDR: An Open Benchmark Evaluating Research Agents That Must Search Wide And Deep appeared first on MarkTechPost.