Driving · 2026-10-09
DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training
Junyan Li, Ruizhi Li, Yu Liu, Xiangshuo Liu, Mingchao Sun, Hongyu Pan, Mu Xu, Lue Fan, Zhaoxiang Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DreamTrue is a multi-view, cross-embodiment robot world model for action-faithful, physically plausible video prediction. It uses offline geometric calibration to align rendered action conditions with recorded videos, then counterfactual post-training with a human-annotated defect dataset and an embodied video reward model to guide RL toward more plausible interactions. On AgiBot, it reports state-of-the-art action following and reduces human-assessed interaction defects from 48.12% to 6.25%.
Why it matters
Action-faithful video prediction is needed for robot learning and evaluation, but dataset calibration errors can make action conditions inconsistent, and datasets biased toward successful outcomes can produce optimistic hallucinations. DreamTrue addresses both using (1) offline geometric calibration without dedicated calibration sequences and (2) counterfactual action post-training guided by a defect-based reward learned from human annotations, enabling feedback under actions beyond recorded trajectories.
Method
- Offline geometric calibration: render a URDF using recorded robot states, build pixel correspondences (robot render/observations, static background across time, synchronized views), then jointly refine camera intrinsics/extrinsics/distortion and optional arm mounting offsets using these constraints.
- Stage I supervised training: represent actions in a unified image-space format by rendering robot RGB/depth/amodal masks and encoding gripper states and camera rays; train a multi-view generator conditioned on these representations across multiple robot datasets.
- Stage II counterfactual post-training: construct counterfactual action trajectories via SE(3) perturbations and inverse kinematics; train a human-annotated embodied video reward model from robot/object/interaction defect labels and use it to guide reward-guided RL post-training.
Abstract (from arXiv)
We present DreamTrue, a multi-view, cross-embodiment robot world model for action-faithful and physically plausible video prediction. Training such a model on existing robot datasets faces two obstacles: imprecise calibration can impair action following, while limited coverage of unsuccessful interactions can bias predictions toward successful outcomes. To improve action following across embodiments, we render action trajectories into image-space conditions and introduce offline geometric calibration to align these conditions with the target videos. To broaden interaction coverage, we introduce counterfactual post-training, modifying recorded action trajectories and generating future videos under a wider range of actions and contact configurations. To provide feedback on these predictions without paired ground-truth futures, we construct a human-annotated video dataset covering robot, object, and interaction defects and use it to train an embodied video reward model. Its scores guide reinforcement-learning post-training toward more physically plausible interaction outcomes. On AgiBot, DreamTrue attains state-of-the-art action following, while reducing the human-assessed interaction defect rate from from 48.12% to 6.25%. Notably, our model ranks first in the world model track of the AgiBot World Challenge 2026. The project page can be found at https://brave-eai.github.io/DreamTrue.
Related papers
- Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling
- Learning Transferable World-Action Models from Task-Paired Human Videos
- Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
- OpenWAM: An Open Framework for Composable World-Action Models
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-09 UTC
DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training
Preprint posted: 2026-10-08 · Original record ↗
- arxiv
- 2610.12468
Explicit source versions (1)
- v1 · 2026-10-08
- Preprint available
Source date: 2026-10-08 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
We present DreamTrue, a multi-view, cross-embodiment robot world model for action-faithful and physically plausible video prediction. Training such a model on existing robot datasets faces two obstacles: imprecise calibration can impair action following, while limited coverage of unsuccessful interactions can bias predictions toward successful outcomes. To improve action following across embodiments, we render action trajectories into image-space conditions and introduce offline geometric calibration to align these conditions with the target videos. To broaden interaction coverage, we introduce counterfactual post-training, modifying recorded action trajectories and generating future videos under a wider range of actions and contact configurations. To provide feedback on these predictions without paired ground-truth futures, we construct a human-annotated video dataset covering robot, object, and interaction defects and use it to train an embodied video reward model. Its scores guide reinforcement-learning post-training toward more physically plausible interaction outcomes. On AgiBot, DreamTrue attains state-of-the-art action following, while reducing the human-assessed interaction defect rate from from 48.12% to 6.25%. Notably, our model ranks first in the world model track of the AgiBot World Challenge 2026. The project page can be found at https://brave-eai.github.io/DreamTrue.