Robotics · 2026-10-14

From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction

Ruo Chen, Jiahua Pang, Yingjie Niu, Hongyi Lin, Jun Tian, Yang Zhu, Chao Li

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DR-WM is a physically consistent world model for quadruped action consequence prediction. From an initial RGB frame, it reconstructs geometry and predicts dense, normalized priors over terrain contact properties (friction, roughness, stiffness). Calibrated with real action–response data, it simulates robot motion with spatially varying contact, derives camera trajectories, and guides video generation. Generated observations are used to iteratively update the terrain for long-horizon rollouts and VLM action selection.

Why it matters

The authors aim to make video futures physically realizable for embodied navigation, since identical actions can lead to different outcomes due to terrain and contact variation. DR-WM grounds video prediction in executable physics by coupling robot–terrain interaction (via calibrated contact dynamics) with video generation and iterative terrain updates, improving both motion prediction metrics and VLM-guided action selection.

Method

Reconstruct scene geometry and learn dense, dimensionless priors over terrain contact properties via PhyMap; map priors to executable parameters calibrated from real robot action–response data.
For each candidate action sequence, simulate robot motion with calibrated, spatially varying contact (Genesis) to obtain base/camera trajectories; condition a video model to generate future observations.
Use generated future frames to update previously unobserved regions of terrain reconstruction, enabling iterative long-horizon causal rollouts without additional real-world images; use VLM rollouts for action selection.

Limitation

The paper states that reliable prediction depends on the accuracy of reconstruction, contact modeling, and generated observations; because generated observations update terrain reconstruction, prediction errors may propagate across successive rollouts, motivating improved geometric consistency and contact modeling over longer horizons.

Abstract (from arXiv)

In outdoor environments, varying terrain and contact conditions can cause identical action commands to produce different motion outcomes for quadruped robots. Predicting these outcomes is essential for reliable navigation and informed path selection. Existing video world models can generate visually plausible futures, yet often struggle to capture robot dynamics and terrain interactions, leading to overly optimistic predictions of action outcomes. We present DR-WM, a physically consistent world model that incorporates robot-terrain contact dynamics to predict the consequences of quadruped actions. Starting from an initial RGB image, our method reconstructs scene geometry and integrates dense, dimensionless priors over terrain contact properties, calibrated using real-world action-response data. The resulting terrain model supports the simulation of physics with spatially varying contact properties such as friction. Given the initial state of the robot and candidate action sequences, we simulate its future motion and use the resulting camera trajectories and rendered geometry to guide the training of a video world model. The generated future observations are then used to update the terrain reconstruction, enabling iterative model rollouts without additional real-world images. Extensive experiments demonstrate that DR-WM substantially outperforms geometry-only baselines and baselines with uniform physical parameters on key physical metrics characterizing action outcomes, while achieving the best performance among the evaluated baselines. These physically grounded predictions support more informed action selection: across 90 challenging scenarios, an 8B vision-language model augmented with DR-WM outperforms an unassisted 32B vision-language model on five of six evaluation metrics.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-14 UTC

preprint