Robotics · 2026-10-16

Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

Zirui Wu, Chen Guo, Tianjian Jiang, Zeren Jiang, Zeming Li, Lingchen Yang, Longteng Duan, Ying Xue, Handi Yin, Shenyuan Gao, Martin R. Oswald, Jun Zhang, Ping Tan, Jie Song

arXiv:2610.22363 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

P3-WAM is a unified world action model for egocentric dexterous manipulation with three modes: Perceive (reconstruct 3D hand motion and camera trajectories from video), Predict (generate action-conditioned visual futures), and Plan (jointly generate action trajectories and their anticipated visual consequences from an initial image, language instruction, and optional visual goal). It integrates compact structured 3D action into a pretrained video diffusion transformer via mixture-of-transformers and randomized modality visibility.

Why it matters

The authors connect perception, action-conditioned prediction, and goal-directed planning in a single model for egocentric manipulation, using unified training that improves each mode and yields higher VLM scores for planning videos. The approach also supports goal-image conditioning for long-horizon tasks and reports robot deployment from planned human motion to dexterous robot commands.

Method

Unifies Perceive/Predict/Plan by training one model with randomized modality visibility: video and actions can be observed or missing to turn the same architecture into reconstruction, prediction, or joint action–future planning.
Integrates structured 3D action tokens (articulated hand + camera states) into a pretrained video diffusion transformer using a mixture-of-transformers architecture with shared base weights, modality-specific low-rank adaptation, and bidirectional visual–action attention.
Plans with language instruction and an optional visual goal image; long tasks use successive visual subgoals per planning window.

Limitation

Estimating the global 3D hand position from monocular video is inherently ambiguous. Incorporating tactile or depth sensing could provide stronger geometric constraints. Their robot experiments currently rely on open-loop execution; closed-loop feedback could improve robustness over longer-horizon interactions.

Abstract (from arXiv)

We introduce P$^{3}$-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-15 · Indexed by Pub-AI: 2026-10-16 UTC

preprint