The causality test that humbled six AI agents

A brand new benchmark out of the University of Michigan simply handed six frontier AI agents a causal reasoning examination. The scores expose a niche that uncooked functionality metrics are inclined to paper over.

The benchmark is named CausalDS, revealed on arXiv on July 9, 2026, by Andrej Leban and Yuekai Sun. It exams one thing extra particular than intelligence.

Can these agentic methods cause about trigger and impact, or do they simply sound prefer it?

The timing is nice, if a bit inconvenient for the business…

Every vendor markets its agents as data-science collaborators as of late. CausalDS asks the tougher query beneath that pitch:

When the reply depends upon separating correlation from intervention, does the agent truly get there?

Why information science agents hold failing causal reasoning

Leban and Sun level to a break up in present benchmarks. Symbolic causal reasoning exams, like CLadder, test a mannequin’s logic over an summary graph.

Data-science agent benchmarks, like DSBench and MLE-bench, test a mannequin’s capacity to jot down code and produce a defensible end result. 

Very few exams cowl each without delay.

The ones constructed from curated real-world datasets additionally carry a contamination threat. A mannequin can rating properly by recalling a well-known instance as an alternative of reasoning by means of the precise construction.

Researchers have a beautiful identify for that shortcut:

The “causal parrot” drawback.

CausalDS closes the hole by producing every little thing from scratch.

Each scene begins with a randomly sampled structural causal mannequin, then the researchers generate observational information from it.

An LLM interprets the summary graph into a site story, full with named variables and models.

A farming subsidy, an irrigation resolution, a soil erosion consequence, all invented for the event and dressed up convincingly. The graph is actual and stays hidden. An audit loop retains the true reply locked away, so early guessing earns a mannequin precisely zero credit score.

Inside the examination

From every scene, the benchmark derives duties spanning all three rungs of Pearl’s causal hierarchy: affiliation, intervention, and counterfactual reasoning.

An agent may predict an consequence from uncooked data, resolve if a remedy impact is even estimable, or flag a proposed adjustment set as biased.

The researchers additionally added a data-quality dial. Some scenes hand the agent clear measurements; others swap key variables for noisy proxies and make it work for the reply.

That separates a mannequin’s grasp of causal construction from its capacity to outlive the messy information each working analyst offers with day by day.

The leaderboard

On a 100-scene real looking examination drawn from a pool of 953 whole scenes, Claude Opus 4.8 topped the desk. It landed a CausalDS rating of 0.278 and an 82.4percent cross charge.

Gemini 3.1 Pro positioned second at 0.370.

Qwen 3.6 35B and Kimi K2.6 adopted, then GPT-5.5, with Gemma 4 26B trailing the sphere at 0.644. The full board, for anybody who likes their proof in desk kind:

• Claude Opus 4.8: rating 0.278, 82.4% cross charge, 17.7k tokens per process
• Gemini 3.1 Pro: rating 0.370, 76.5% cross charge, 145.6k tokens per process
• Qwen 3.6 35B: rating 0.447, 63.2% cross charge, 140.7k tokens per process
• Kimi K2.6: rating 0.475, 65.7% cross charge, 266.4k tokens per process
• GPT-5.5: rating 0.561, 82.4% cross charge, 12.9k tokens per process
• Gemma 4 26B: rating 0.644, 55.9% cross charge, 32.4k tokens per process

The token column is the place issues get genuinely humorous.

Claude Opus 4.8 reached its main rating on 17.7k tokens per process, whereas Kimi K2.6 burned by means of 266.4k tokens for a worse end result, a fifteen-fold hole for much less to point out for it.

GPT-5.5 matched the chief’s 82.4% cross charge utilizing the fewest tokens on the board, but completed fifth general. Efficient reasoning and correct reasoning are apparently two separate expertise, a distinction most benchmark advertising and marketing tends to gloss over.

Gemma 4 26B deserves its personal paragraph. It submitted a legitimate reply on each single process, an ideal completion charge, and nonetheless completed final. In informal work, that mixture normally means one factor: the mannequin answered when the wiser transfer was to pause, just like the colleague who fills each silence in a gathering with assured nonsense.

Five axes, one rating

CausalDS scores each agent throughout 5 separate capabilities, and the paper’s central discovering is that these transfer independently. A mannequin robust on one axis can land weak on one other, which is precisely why a single leaderboard quantity invitations the fallacious conclusion.

• Symbolic causal reasoning: separating associational claims from interventional and counterfactual ones.
• Data-science execution: writing and operating code that truly produces an estimate from tabular information.
• Uncertainty quantification: attaching a calibrated interval to an estimate as an alternative of a falsely exact level worth.
• Epistemic abstention: recognizing when the information genuinely fall in need of a solution, then abstaining moderately than hallucinating a assured one.
• Tool use and coding effectivity: reaching an accurate reply inside a sandboxed setting whereas preserving steps and token spend in test.

What this implies for groups evaluating agentic AI

Any group piloting agents for causal inference, advertising and marketing combine modeling, or root-cause evaluation ought to deal with CausalDS as a preview of the failure modes ready in production.

An agent that all the time solutions alerts confidence, however confidence and reliability are totally different currencies. A vendor demo that solely rewards uncooked accuracy tends to cover that distinction from the folks making the buying selections.

Leban and Sun body this properly within the paper. A associated benchmark, CausaLab, exams an experimental discoverer.

CausalDS exams a real causal data scientist working from a pile of information and a deadline. That is the function most enterprise groups really want crammed.

Just a few questions price asking earlier than trusting an agent with an actual causal declare:

• Ask distributors for cross charge and abstention charge individually, moderately than a single blended accuracy determine a confidently fallacious mannequin can inflate.
• Test agents in opposition to intentionally non-identifiable questions, the causal equal of a trick query, earlier than trusting output on ambiguous real-world information.
• Track token value per right reply. CausalDS exhibits some fashions pay for accuracy with budgets many instances bigger than a rival hitting an analogous cross charge.