Verifiable causal discovery benchmark for autonomous experiment design
Use a sandbox benchmark that supplies configurable structural equation models with explicit resource constraints, then evaluate an agent’s ability to plan experiments, iterate through sequential observations, analyze results, and decide when to stop and conclude—without providing domain-specific semantic cues.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
It separates failures from causal inference alone versus failures arising from how experiments are designed and executed, making it easier to target improvements to experimental planning and sequential decision-making rather than reasoning over evidence in isolation.
What changed
The benchmark introduces verifiable, configurable experimental environments (structural equation models) with resource limits and sequential interaction episodes, enabling action-trajectory-based analysis of where mistakes originate; it also reports that simpler statistical tools can help more than advanced tools when agents lack foundational capabilities.
Where it helps
- Evaluating LLM agents that must design sequential experiments for causal discovery
- Debugging whether errors come from experimental design vs causal inference reasoning
- Comparing the effect of different statistical tooling on autonomous causal discovery
- Assessing stopping rules and resolution of conflicting evidence in sequential trials
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
Findings are tied to the benchmark’s specific experimental setting (structural equation models in the sandbox, absence of domain-semantic cues) and the reported agent/tool behaviors observed within those conditions.
Sources
- Causal Discovery Bench: Can LLMs Autonomously Conduct Effective Experiments? ↗
arXiv · primary · Checked 2026-11-10
