Robotics · 2026-10-01

Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling

Xiangyu Zhu, Jin Xu, Yue Guo, Xin Wu, Yifan Sun, Xiancong Ren, Jianxin Sun, Yong Dai, Xiaozhu Ju

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Dream4ACT is a video–action world model for multi-embodiment joint control. It unifies heterogeneous joint spaces using “action views”: URDF-based forward-kinematics renderings of target joint configurations from four fixed virtual cameras. A shared VAE+diffusion transformer models RGB observations and four action-view streams with masked flow matching for forward dynamics, inverse dynamics, and joint generation; predicted views are converted to executable joints via training-free URDF-constrained multiview recovery.

Why it matters

The authors target a mismatch between joint-space action vectors (varying across embodiments) and the image-space priors of video generation models. By encoding target robot configurations as a fixed multiview visual interface, they enable shared tokenization and video modeling across embodiments, while still recovering embodiment-executable joint targets without a learned embodiment-specific action decoder.

Method

Represents target joint configurations as four fixed multiview “action views” rendered via URDF-based forward kinematics, using shared camera roles across embodiments.
Trains a shared video VAE and diffusion transformer on physical-camera RGB observations plus action-view streams, with masked flow matching supporting forward dynamics, inverse dynamics, and joint generation within one model.
Recovers executable joint targets from predicted action views using a training-free, URDF-constrained multiview Gauss–Newton matching procedure (no learned per-robot action decoder).

Limitation

Their demonstrated scope is bounded by training data and embodiment coverage (evaluation includes five simulated embodiments and four real-world platforms); broader task coverage and transfer to unseen embodiments remain to be validated. Recovery also requires embodiment-specific URDF and rendering conventions consistent with training, and explicit multiview generation plus numerical action recovery incurs substantial latency (about 6.9 s and 2.7 s per 80-step chunk on real hardware), so it supports chunk-based execution rather than real-time per-step replanning.

Abstract (from arXiv)

Video generation models (VGMs) offer strong spatiotemporal priors for embodied observation--action modeling. However, joint-space action vectors lack explicit image-space structure and vary in dimensionality and semantics across embodiments, making it challenging to directly leverage the rich spatiotemporal priors of VGMs. End-effector visualizations provide an alternative but do not specify the full articulated configuration needed for robot execution. We present Dream4ACT, a world model built for joint video-action modeling across embodiments. To unify action representations across embodiments, we introduce a shared visual action interface, called action views, which render target joint configurations from four prescribed virtual cameras using URDF-based forward kinematics. This shared visual representation preserves embodiment-specific articulated geometry while allowing observation and action sequences to share a video autoencoder and diffusion transformer. Through masked flow-matching, our model supports forward dynamics, inverse dynamics, and joint observation--action generation within a single jointly trained model by varying which future sequences are corrupted. To recover executable action sequences from predicted action views, we propose a training-free, URDF-constrained multiview recovery mechanism, without a learned embodiment-specific decoder. Dream4ACT achieves an average success rate of 88.98\% on RoboTwin~2.0 and an overall score of 65.66 on TriWorldBench, supporting effective closed-loop manipulation and competitive action-conditioned multiview prediction through the visual action interface.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC

preprint