Driving · 2026-10-08
RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
Zhiqin Yang, Chenxin Li, Xiaomeng Hu, Yibin Liu, Weidong Huang, Jiankai Sun, Haitao Li, Zijian Wu, Yuzhi Huang, Fanding Huang, Hanwen Sun, Jiashun Liu, Jingqi Tong, Mingxin Huang, Shaoli Hu, Shijue Huang, Tianyi Bai, Xinyuan Wang, Yunlong Lin, Zhengyang Tang, Zhexin Zhang, Zhuo Chen, Xierui Song, Juntao Dai, Boyuan Chen, Jiaming Ji, Fangneng Zhan, Mengkang Hu, Wei Xue, Yonggang Zhang, Han Hu, Tsung-Yi Ho, Yike Guo
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
RobotWorld is a simulation benchmark for “robot use”: turning instructions and observations into physical task execution through robot interfaces. It evaluates 84 tasks across manipulation, mobile manipulation, locomotion, driving, and aerial control with explicit interaction budgets and executable success checks. The authors analyze outcomes with execution traces to identify both transferable capabilities (perception/control workflows) and persistent gaps that prevent reliable task completion.
Why it matters
The authors argue that digital agent abilities do not reliably compose into successful physical behavior, and that progress needs measurable capability gaps. RobotWorld provides a common evaluation protocol (tasks, budgets, success checkers, recorded traces) across diverse embodiments and control demands, and reports model-dependent strengths and failure modes tied to execution behavior.
Method
- Introduce RobotWorld, a simulation benchmark with 84 robot-use tasks across heterogeneous embodiments and control demands, using explicit task contracts (observation/action interfaces, native controllers, budgets, executable success checkers).
- Use an execution harness that alternates analysis (paused physics) with bounded robot execution; optionally supports code control with feedback at each control step.
- Analyze task outcomes alongside execution traces to characterize both capability transfers (e.g., segmentation, calibration, spatial estimation, dynamics-based computation) and failure mechanisms (state interpretation, action revision timing, recovery, completion assessment).
Limitation
The authors state that their trajectory and example analyses are not turned into corpus-wide frequency estimates, and that matched-task comparisons are observational snapshots that can have confounders from provider/collection times and incomplete build metadata.
Abstract (from arXiv)
General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world. To investigate this, we introduce RobotWorld, a challenging simulation testbed for robot use: turning instructions and observations into physical task execution through robot interfaces. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving, and aerial control, with explicit interaction budgets and executable success checks. By analysing task outcomes alongside execution traces, we identify both the capabilities that transfer and the gaps that prevent reliable completion. Furthermore, we find that current agents can construct sophisticated perception and control workflows, including image segmentation, camera calibration, spatial estimation, and dynamics-based computation. These capabilities, however, do not consistently compose into successful behaviour: agents lose task-relevant object states despite reaching commanded poses, fail to correct ineffective actions, recover too late, or mistake unfinished tasks for completion. This uneven transfer also differs across models: Astra succeeds more often on spatial and constrained-contact goals, whereas Opus 5.5 succeeds more often on continuous-balance and timed-interaction goals. By linking these outcomes to execution behaviour, RobotWorld provides both a rigorous proving ground and an empirical account of the remaining capability gaps, thereby establishing concrete targets for training and designing more reliable physical-world agents.
Related papers
- Explore, Execute, Evolve: A Skill Acquisition and Reuse Loop for Embodied Agents
- RoboBridge: A Self-Evolving Embodied Agent Framework for Sim-to-Real Transfer
- Towards Agentic Robots: Model, Data, Environment, and Harness
- Agentic World Action Models
- Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC
RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
Preprint posted: 2026-10-07 · Original record ↗
- arxiv
- 2610.10409
Explicit source versions (1)
- v1 · 2026-10-07
- Preprint available
Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world. To investigate this, we introduce RobotWorld, a challenging simulation testbed for robot use: turning instructions and observations into physical task execution through robot interfaces. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving, and aerial control, with explicit interaction budgets and executable success checks. By analysing task outcomes alongside execution traces, we identify both the capabilities that transfer and the gaps that prevent reliable completion. Furthermore, we find that current agents can construct sophisticated perception and control workflows, including image segmentation, camera calibration, spatial estimation, and dynamics-based computation. These capabilities, however, do not consistently compose into successful behaviour: agents lose task-relevant object states despite reaching commanded poses, fail to correct ineffective actions, recover too late, or mistake unfinished tasks for completion. This uneven transfer also differs across models: Astra succeeds more often on spatial and constrained-contact goals, whereas Opus 5.5 succeeds more often on continuous-balance and timed-interaction goals. By linking these outcomes to execution behaviour, RobotWorld provides both a rigorous proving ground and an empirical account of the remaining capability gaps, thereby establishing concrete targets for training and designing more reliable physical-world agents.