Driving · 2026-10-26
WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations
Tianling Xu, Weijie Wan, Honghao Su, Kailun Su, Junyuan Tang, Huichi Zhou, Haozheng Zhang, Qi Xiong, Ping Luo, Wenbo Ding, Tianxing Chen, Yue Chen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
WorldMirror is an agentic real-to-sim framework that turns in-the-wild robot demonstrations (RGB video + trajectories) into simulation-ready interactive digital twins. It formulates construction as iterative refinement of an executable world program, using a camera-pose-anchored scene representation and verification-driven repair, enabling direct trajectory replay without motion replanning or trajectory retargeting.
Why it matters
The authors address the need for interactive simulation environments that preserve real-world appearance and physically consistent interactions without heavy manual scene/task construction. WorldMirror aims to produce digital twins where the scene, objects, and robot share a camera-anchored coordinate frame so recorded robot trajectories remain executable, and where verification feedback drives targeted reconstruction fixes for better replay and interaction fidelity.
Method
- Iterative refinement of an executable world program: a coding agent decomposes reconstruction, orchestrates specialized image/3D generation tools, and refines the program using verification feedback.
- Camera-pose-anchored scene representation: organizes reconstructed worlds into static visual context, support geometry, and editable interactive assets while preserving spatial relationships; supports executing recorded trajectories without retargeting.
- Router-based adaptive orchestration with tool delegation: a router selects legal graph transitions (advance/rewind/inspect/stop) based on typed verification failures; specialist updates are committed only after passing checks.
Limitation
The authors note: “Our evaluation remains limited to a small set of DROID demonstrations and does not yet include downstream policy learning, real-world deployment, or broader cross-task generalization. The reconstruction quality also depends on the available reconstruction and simulation tools.”
Abstract (from arXiv)
Existing approaches for robot learning in simulation often require substantial manual effort to construct scenes and specify tasks, limiting training scalability and data diversity. Meanwhile, in-the-wild robot demonstrations naturally provide diverse and realistic observations of real-world interactions. Leveraging these demonstrations for simulation requires both high-fidelity visual reconstruction and physically consistent interactions. General-purpose agents offer flexible adaptation to diverse scenarios and strong reasoning about 3D spatial relationships, while specialized generative models provide high-fidelity reconstruction of scene appearance and geometry. We introduce WorldMirror, an agentic real-to-sim framework that constructs interactive digital twins from in-the-wild robot demonstrations. Our central idea is to formulate real-to-sim construction as the iterative refinement of an executable world program. A coding agent decomposes the observed scene, plans reconstruction steps, and orchestrates specialized image and 3D generation tools, refining the program through verification feedback. A camera-pose-anchored scene representation organizes the world into static visual context, support geometry, and editable interactive assets while preserving spatial relationships among the scene, objects, and robot. The agent uses this representation to diagnose and correct reconstruction errors, enabling physically consistent interaction and trajectory replay. Experiments on DROID demonstrate that WorldMirror can reconstruct visually faithful and geometrically consistent interactive worlds while preserving the physical validity of recorded robot trajectories.
Related papers
- Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation
- Code2Worlds-2: Physically Plausible Agent Harness for Embodied World Simulation
- Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
- Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
- BWM-Forge: Agentic World Programming for Embodied Intelligence
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-26 UTC
WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations
Preprint posted: 2026-10-22 · Original record ↗
- arxiv
- 2610.31834
Explicit source versions (1)
- v1 · 2026-10-22
- Preprint available
Source date: 2026-10-22 · First detected: 2026-11-02 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Existing approaches for robot learning in simulation often require substantial manual effort to construct scenes and specify tasks, limiting training scalability and data diversity. Meanwhile, in-the-wild robot demonstrations naturally provide diverse and realistic observations of real-world interactions. Leveraging these demonstrations for simulation requires both high-fidelity visual reconstruction and physically consistent interactions. General-purpose agents offer flexible adaptation to diverse scenarios and strong reasoning about 3D spatial relationships, while specialized generative models provide high-fidelity reconstruction of scene appearance and geometry. We introduce WorldMirror, an agentic real-to-sim framework that constructs interactive digital twins from in-the-wild robot demonstrations. Our central idea is to formulate real-to-sim construction as the iterative refinement of an executable world program. A coding agent decomposes the observed scene, plans reconstruction steps, and orchestrates specialized image and 3D generation tools, refining the program through verification feedback. A camera-pose-anchored scene representation organizes the world into static visual context, support geometry, and editable interactive assets while preserving spatial relationships among the scene, objects, and robot. The agent uses this representation to diagnose and correct reconstruction errors, enabling physically consistent interaction and trajectory replay. Experiments on DROID demonstrate that WorldMirror can reconstruct visually faithful and geometrically consistent interactive worlds while preserving the physical validity of recorded robot trajectories.
