Driving · 2026-10-20
UMI-WM: Interactive World Modeling for Bimanual Manipulation from Wrist Cameras
Jingxiang Guo, Haoyu Zhao, Xidong Zhang, Yuhang Zheng, Hao Wu, Chen Gao, Jianshu Zhou, Shuicheng Yan, Weihao Yuan
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
UMI-WM is an action-conditioned bimanual interactive world model that jointly generates paired left/right wrist-camera videos from observation history, per-arm relative camera poses, and gripper widths. It uses view-specific action injection plus sparse hub tokens for cross-view scene exchange, and camera-aware geometry conditioning with geometry-guided noise. A 30-step Teacher is distilled into a four-step Student for block-causal rollout.
Why it matters
The authors target bimanual manipulation from wrist cameras, emphasizing cross-view consistency and action following for two independently controlled end-effectors observing a shared scene. They propose an evaluation that recovers per-arm relative camera poses and gripper widths from generated videos via an inverse dynamics model (IDM) and compute Bimanual Action Fidelity (BAF) to measure agreement with conditioning actions.
Method
- Action-conditioned dual-stream generation: each wrist stream receives its own per-arm action state; sparse hub tokens exchange shared-scene information between streams.
- Camera-aware geometry conditioning: camera-guided noise warping and ray/relative-pose-based attention biases support per-view motion control and cross-view geometric exchange.
- Few-step causal generation: adapt the model with block-causal attention and distill a 30-step Teacher into a four-step Student for interactive rollout.
Abstract (from arXiv)
Universal Manipulation Interface (UMI) pairs wrist-camera videos with camera trajectories and gripper openings, providing scalable, robot-free data for learning how bimanual actions change a shared scene. Existing interactive world models, however, typically rely on text or single-camera control and struggle to follow each arm's actions while maintaining consistency between the two wrist views. We introduce UMI-WM, an action-conditioned bimanual world model that jointly generates both wrist views from observation history, per-arm relative camera poses, and gripper widths, with cross-view consistency, action following, and visual fidelity. UMI-WM injects each arm's actions into its own wrist stream, while sparse hub tokens exchange scene information between the two streams. Camera-aware geometry conditioning and camera-guided noise further support per-view motion control and cross-view geometric exchange. We adapt the model for block-causal generation and distill the 30-step Teacher into a Student with four denoising steps. We train UMI-WM on self-collected open-world and in-studio UMI recordings combined with public manipulation data. To evaluate action following, we use an inverse dynamics model (IDM) to recover relative camera poses and gripper widths from generated videos, and introduce Bimanual Action Fidelity (BAF) to measure how closely these recovered actions match the conditioning actions. UMI-WM achieves the best results on all five reference-video fidelity metrics and the best or competitive performance on VBench and WorldArena. The Teacher and Student outperform all compared baselines in action following, while the four-step Student generates paired wrist videos 6.9 times faster than the Teacher.
Related papers
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation
- PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
- Spatial-UMI-Dream: Scaling World Action Models with Metric Geometry for UMI Manipulation
- BAT-WAM: Bridging Action and Touch in World-Action Models for Dexterous Hands
- Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-20 UTC
UMI-WM: Interactive World Modeling for Bimanual Manipulation from Wrist Cameras
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.23913
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Universal Manipulation Interface (UMI) pairs wrist-camera videos with camera trajectories and gripper openings, providing scalable, robot-free data for learning how bimanual actions change a shared scene. Existing interactive world models, however, typically rely on text or single-camera control and struggle to follow each arm's actions while maintaining consistency between the two wrist views. We introduce UMI-WM, an action-conditioned bimanual world model that jointly generates both wrist views from observation history, per-arm relative camera poses, and gripper widths, with cross-view consistency, action following, and visual fidelity. UMI-WM injects each arm's actions into its own wrist stream, while sparse hub tokens exchange scene information between the two streams. Camera-aware geometry conditioning and camera-guided noise further support per-view motion control and cross-view geometric exchange. We adapt the model for block-causal generation and distill the 30-step Teacher into a Student with four denoising steps. We train UMI-WM on self-collected open-world and in-studio UMI recordings combined with public manipulation data. To evaluate action following, we use an inverse dynamics model (IDM) to recover relative camera poses and gripper widths from generated videos, and introduce Bimanual Action Fidelity (BAF) to measure how closely these recovered actions match the conditioning actions. UMI-WM achieves the best results on all five reference-video fidelity metrics and the best or competitive performance on VBench and WorldArena. The Teacher and Student outperform all compared baselines in action following, while the four-step Student generates paired wrist videos 6.9 times faster than the Teacher.
