Driving · 2026-10-14
WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions
Haoyi Duan, Liliang Chen, Nan Wang, Yang Zhang, Yuhang Lu, Xiaotong Chen, Wei-Cheng Tseng, Shenyuan Gao, Zhaoshuo Li, Jingyi Jin, Pooya Jannaty, Dhruv Shah, Yuke Zhu, Ming-Yu Liu, Maoqing Yao, Guanghui Ren
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
WorldOracle is a benchmark that tests action-conditioned world models by comparing predicted open-loop rollouts to paired real-world executions using the same initial observation and recorded low-level action sequence. The authors evaluate execution coherence along physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence, using long multi-segment episodes with successful and failed interactions. They release datasets, evaluation code, and a leaderboard.
Why it matters
The benchmark targets “execution coherence”: whether a predicted rollout reproduces the physical consequences and task-state transitions of the same action trajectory in the real world, including failed attempts over long horizons. It fills a gap where existing embodied world-model benchmarks largely evaluate visual quality or simulation-based executability rather than matching paired real-world action-conditioned outcomes.
Method
- Benchmark question: given the same initial observation and recorded low-level action sequence, does an ACWM’s predicted rollout match the paired real-world execution?
- Evaluate predicted rollouts on four dimensions: physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence.
- Curate real-world training/evaluation suites across three embodiments (AgiBot G2, YAM, human-hand) with long multi-segment episodes containing both successful and failed executions; provide a VLM-assisted annotation pipeline and a leaderboard.
Limitation
Single real-world reference. Each trial uses one observed real-world execution as a reference. WorldOracle may penalize alternative outcomes that are physically plausible under stochastic contact, fluid, or deformable-object dynamics.
Abstract (from arXiv)
We introduce WorldOracle, a benchmark specifically designed to evaluate action-conditioned world models (ACWMs) against paired real-world executions. Given the same initial observation and low-level action sequence, WorldOracle asks whether a predicted rollout reproduces what happens in the real world, including physical consequences and task-state transitions. We evaluate models along four complementary dimensions: physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence. To support this evaluation, we curate real-world training and evaluation suites that cover different interaction dynamics and embodiments. The evaluation suite contains long, multi-segment episodes with both successful and failed executions, enabling evaluation across varied action outcomes and behavior distributions. Experiments on seven representative ACWMs reveal distinct model strengths and failure modes. WorldOracle provides a systematic foundation for measuring and advancing action-conditioned world modeling for manipulation. We will release the datasets, evaluation code, and leaderboard.
Related papers
- UniW(A)M: A Unified Mixture-of-Experts Framework for World and World-Action Models
- Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation
- What Limits Cross-Task Generalization in World Action Models?
- Plausible Future, Wrong Choice: Steering World-Action Models toward Intended Future
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-14 UTC
WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions
Preprint posted: 2026-10-13 · Original record ↗
- arxiv
- 2610.18826
Explicit source versions (1)
- v3 · 2026-10-21
- Preprint available
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
- Explicit source version detected · v3
Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
Abstract · arxiv
We introduce WorldOracle, a benchmark specifically designed to evaluate action-conditioned world models (ACWMs) against paired real-world executions. Given the same initial observation and low-level action sequence, WorldOracle asks whether a predicted rollout reproduces what happens in the real world, including physical consequences and task-state transitions. We evaluate models along four complementary dimensions: physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence. To support this evaluation, we curate real-world training and evaluation suites that cover different interaction dynamics and embodiments. The evaluation suite contains long, multi-segment episodes with both successful and failed executions, enabling evaluation across varied action outcomes and behavior distributions. Experiments on seven representative ACWMs reveal distinct model strengths and failure modes. WorldOracle provides a systematic foundation for measuring and advancing action-conditioned world modeling for manipulation. We release the datasets, evaluation code, and leaderboard: https://worldoracle-benchmark.github.io/.