Driving · 2026-10-05

DeltaWorld: Physically Consistent Interactive World Simulators via Action-Conditioned Latent Increment Learning

Boyuan Hou, Xiaoge Cao, Chaofan Zhang, Shuo Wang, Shaowei Cui

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DeltaWorld is an action-conditioned interactive world simulator for robotic manipulation that predicts action-induced latent increments rather than the full next latent state. It uses Delta-LTM to add predicted latent feature changes to the current latent state for long-horizon rollouts, and Interaction-aware Latent Alignment to supervise counterfactual interaction regions to reduce physically implausible outcomes.

Why it matters

The authors target physically consistent, action-faithful long-horizon video prediction for robotics, motivated by failures of methods that directly predict entire next latent states (e.g., object interpenetration or excessive deformation). DeltaWorld explicitly models small action-induced changes via latent increments and emphasizes robot–object interaction regions during training.

Method

Delta Latent Transition Model (Delta-LTM): predict action-conditioned latent feature increments Δẑt and update the next latent as ẑt+1 = zt + Δẑt (residual update), using an action-conditioned 3D conv consistency dynamics model with FiLM conditioning.
Interaction-aware Latent Alignment: construct counterfactual interaction regions by removing robot arm and manipulated object regions, compute latent differences, threshold to get an interaction mask, and apply localized supervision on latent increment residuals.
Two-stage training: Stage 1 trains a frozen encoder/decoder autoencoder with consistency-based conditional decoding and latent noise injection; Stage 2 freezes encoder/decoder and optimizes only the dynamics model on action-conditioned increments using global and interaction-masked losses.

Abstract (from arXiv)

Interactive world simulators can provide scalable environments for robot planning, policy training, and evaluation by predicting action consequences while reducing reliance on repeated physical rollouts. To serve these applications, they must generate future image sequences that respond faithfully to robot actions and preserve the dynamics of robot-object interactions over long horizons. However, existing world models typically predict the entire next latent state and often fail to capture subtle changes induced by robot actions. Such omissions can produce physically implausible outcomes, including object interpenetration and excessive deformation. To address this limitation, we propose DeltaWorld, a physically consistent interactive world simulator for robotic manipulation. Our method introduces the Delta Latent Transition Model (Delta-LTM), which predicts action-induced latent feature changes and adds them to the current latent state to obtain the next state, rather than predicting the next latent state directly. To mitigate object interpenetration and excessive deformation in predicted future frames, Interaction-aware Latent Alignment is introduced to construct counterfactual interaction regions and supervise interaction-related latent changes. DeltaWorld is evaluated on the IWS manipulation benchmark and a self-collected cross-robot dataset covering multiple robot embodiments and manipulation tasks. On the cross-robot dataset, DeltaWorld reduces FVD by 46.6% and LPIPS by 31.1% relative to the IWS baseline. These results highlight the potential of DeltaWorld for long-horizon action-conditioned video prediction in robotic manipulation.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC

preprint