Robotics · 2026-10-30
Gym-Anything for Robotics: Scaling physical simulations with digital agents
Pranjal Aggarwal, Sean Welleck
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce Gym-Anything for Robotics, a framework where coding agents create robotics simulations and an adversarial creation-audit loop checks simulations and tasks against defined criteria. They build RoboWorld: 25 simulations with 25 test and 175 training RL environments. Frontier models fail all RoboWorld test tasks (average reward 0.06). The authors propose new learning methods using privileged simulator state and release the pipeline, environments, and benchmark.
Why it matters
The paper targets a key bottleneck in scaling realistic robotics simulation: expert time to build and verify environments. By framing simulation creation as a coding-agent task and using multi-agent adversarial auditing for both simulations and tasks, the authors aim to produce broader, more faithful physical work benchmarks. RoboWorld provides a standardized evaluation and several privileged-state learning setups to study how agents can improve in simulation.
Method
- Gym-Anything for Robotics: coding agents build simulations in code; evidence is audited by multiple audit agents (including adversarial breaking) against criteria for a good simulation and task.
- RoboWorld construction: select simulation domains via U.S. GDP-grounded decomposition into atomic physical actions; tasks are generated with controlled difficulty and verified solvability and reward correctness.
- Privileged-state learning in simulation: models use restricted or privileged access; the paper studies in-context learning, a privileged teacher, self-assisted learning from privileged experience, and privileged code policies to train VLAs (π0.5).
Abstract (from arXiv)
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $\pi_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Related papers
- Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
- EmbodiedEvo: Self-improving Coding Agents for Physical Intelligence
- SEAR: Self-Evolving Agentic Robotics
- Towards Agentic Robots: Model, Data, Environment, and Harness
- Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC
Gym-Anything for Robotics: Scaling physical simulations with digital agents
Preprint posted: 2026-10-29 · Original record ↗
- arxiv
- 2610.40565
Explicit source versions (1)
- v1 · 2026-10-29
- Preprint available
Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40565v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics