Video World Models · 2026-10-12
Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
Feng Jiang, Shu Chen, Yuanfei Wang, Yuan Li, Zhenhao Shen, Jiaqi Liang, Yang Chen, Lei Wei, Chen Xie, Ruihai Wu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Grounded-Dreamer synthesizes physically grounded robot manipulation data using a video world model plus simulator physics. Starting from a small set of real images of task initial states, it reconstructs task-relevant objects and camera geometry, collects grounded trajectories in simulation, post-trains a video world model and an inverse dynamics model, generates manipulation videos from real images, and filters rollouts via closed-loop simulation replay (keeps only task-successful). Policies are trained only on this grounded synthetic data.
Why it matters
The authors address a scaling bottleneck in robot manipulation: obtaining large action-labeled datasets without expensive teleoperation and without the sim-to-real physical/visual mismatch of purely simulated or purely generative rollouts. Their approach combines generative video realism with simulator physics verification, then trains real-world policies solely on filtered synthetic trajectories for out-of-distribution robustness.
Method
- Real-to-sim reconstruction: calibrate camera poses using FEEPE, reuse Lehome assets, reconstruct rigid objects with SAM3D and articulated objects via SAM3D components plus kinematic/joint rules, aligning simulation viewpoints to real FOVs.
- Grounded trajectory collection + post-training: in the aligned simulator, use an LLM-based agent to generate/exeсute data-collection scripts (inverse kinematics control), collecting 100–150 trajectories per task; apply interaction-aware masking to fine-tune Cosmos-Predict 2.5 and the inverse dynamics model (IDM).
- Video synthesis + closed-loop verification: generate rollout videos conditioned on augmented initial frames, use the IDM to extract action trajectories, then replay trajectories in reconstructed simulation and retain only those that successfully complete tasks for policy training.
Limitation
While Grounded-Dreamer avoids human teleoperation, it currently requires a minimal one-time collection of reference images (typically one scene overview and 1–4 object perspectives). The success rate for synthesized initial frames is approximately 65%.
Abstract (from arXiv)
Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale. Existing methods either depend on human teleoperation, rely on simulation with limited visual realism, or use human videos that lack embodiment-specific annotations. In this paper, we introduce Grounded-Dreamer, a framework for synthesizing physically grounded robot manipulation data with video world models while avoiding human teleoperation demonstrations. Grounded-Dreamer combines the visual realism of generative video models with simulator physics engines. Starting from a small set of real images depicting robot task initial states, our method automatically reconstructs task-relevant objects and camera geometry, aligns them in simulation, and collects physically valid trajectories. These trajectories are used to post-train both a video world model and an inverse dynamics model, enabling the generation of visually realistic robot manipulation videos and corresponding robot actions from real images. To filter physically implausible rollouts, Grounded-Dreamer replays generated trajectories in the reconstructed simulation and retains only those that successfully complete the tasks. The real-world robot policies are trained solely on the grounded synthetic trajectories without any human demonstrations. Empirical results in real-world tasks demonstrate that Grounded-Dreamer can synthesize physically grounded data and substantially facilitate policy learning without requiring human teleoperation effort.
Related papers
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
- CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization
- AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning
- RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer
- Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-12 UTC
Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
Preprint posted: 2026-10-09 · Original record ↗
- arxiv
- 2610.13164
Explicit source versions (1)
- v1 · 2026-10-09
- Preprint available
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale. Existing methods either depend on human teleoperation, rely on simulation with limited visual realism, or use human videos that lack embodiment-specific annotations. In this paper, we introduce Grounded-Dreamer, a framework for synthesizing physically grounded robot manipulation data with video world models while avoiding human teleoperation demonstrations. Grounded-Dreamer combines the visual realism of generative video models with simulator physics engines. Starting from a small set of real images depicting robot task initial states, our method automatically reconstructs task-relevant objects and camera geometry, aligns them in simulation, and collects physically valid trajectories. These trajectories are used to post-train both a video world model and an inverse dynamics model, enabling the generation of visually realistic robot manipulation videos and corresponding robot actions from real images. To filter physically implausible rollouts, Grounded-Dreamer replays generated trajectories in the reconstructed simulation and retains only those that successfully complete the tasks. The real-world robot policies are trained solely on the grounded synthetic trajectories without any human demonstrations. Empirical results in real-world tasks demonstrate that Grounded-Dreamer can synthesize physically grounded data and substantially facilitate policy learning without requiring human teleoperation effort.