Driving · 2026-10-20

UMI-WM: Interactive World Modeling for Bimanual Manipulation from Wrist Cameras

Jingxiang Guo, Haoyu Zhao, Xidong Zhang, Yuhang Zheng, Hao Wu, Chen Gao, Jianshu Zhou, Shuicheng Yan, Weihao Yuan

arXiv:2610.23913PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

UMI-WM is an action-conditioned bimanual interactive world model that jointly generates paired left/right wrist-camera videos from observation history, per-arm relative camera poses, and gripper widths. It uses view-specific action injection plus sparse hub tokens for cross-view scene exchange, and camera-aware geometry conditioning with geometry-guided noise. A 30-step Teacher is distilled into a four-step Student for block-causal rollout.

Why it matters

The authors target bimanual manipulation from wrist cameras, emphasizing cross-view consistency and action following for two independently controlled end-effectors observing a shared scene. They propose an evaluation that recovers per-arm relative camera poses and gripper widths from generated videos via an inverse dynamics model (IDM) and compute Bimanual Action Fidelity (BAF) to measure agreement with conditioning actions.

Method

  • Action-conditioned dual-stream generation: each wrist stream receives its own per-arm action state; sparse hub tokens exchange shared-scene information between streams.
  • Camera-aware geometry conditioning: camera-guided noise warping and ray/relative-pose-based attention biases support per-view motion control and cross-view geometric exchange.
  • Few-step causal generation: adapt the model with block-causal attention and distill a 30-step Teacher into a four-step Student for interactive rollout.
Abstract (from arXiv)

Universal Manipulation Interface (UMI) pairs wrist-camera videos with camera trajectories and gripper openings, providing scalable, robot-free data for learning how bimanual actions change a shared scene. Existing interactive world models, however, typically rely on text or single-camera control and struggle to follow each arm's actions while maintaining consistency between the two wrist views. We introduce UMI-WM, an action-conditioned bimanual world model that jointly generates both wrist views from observation history, per-arm relative camera poses, and gripper widths, with cross-view consistency, action following, and visual fidelity. UMI-WM injects each arm's actions into its own wrist stream, while sparse hub tokens exchange scene information between the two streams. Camera-aware geometry conditioning and camera-guided noise further support per-view motion control and cross-view geometric exchange. We adapt the model for block-causal generation and distill the 30-step Teacher into a Student with four denoising steps. We train UMI-WM on self-collected open-world and in-studio UMI recordings combined with public manipulation data. To evaluate action following, we use an inverse dynamics model (IDM) to recover relative camera poses and gripper widths from generated videos, and introduce Bimanual Action Fidelity (BAF) to measure how closely these recovered actions match the conditioning actions. UMI-WM achieves the best results on all five reference-video fidelity metrics and the best or competitive performance on VBench and WorldArena. The Teacher and Student outperform all compared baselines in action following, while the four-step Student generates paired wrist videos 6.9 times faster than the Teacher.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-20 UTC

preprint

UMI-WM: Interactive World Modeling for Bimanual Manipulation from Wrist Cameras

Jingxiang Guo, Haoyu Zhao, Xidong Zhang, Yuhang Zheng, Hao Wu, Chen Gao, Jianshu Zhou, Shuicheng Yan, Weihao Yuan

Preprint posted: 2026-09-30 · Original record ↗

arxiv
2610.23913
Explicit source versions (1)
  • v1 · 2026-09-30
  1. Preprint available

    Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Universal Manipulation Interface (UMI) pairs wrist-camera videos with camera trajectories and gripper openings, providing scalable, robot-free data for learning how bimanual actions change a shared scene. Existing interactive world models, however, typically rely on text or single-camera control and struggle to follow each arm's actions while maintaining consistency between the two wrist views. We introduce UMI-WM, an action-conditioned bimanual world model that jointly generates both wrist views from observation history, per-arm relative camera poses, and gripper widths, with cross-view consistency, action following, and visual fidelity. UMI-WM injects each arm's actions into its own wrist stream, while sparse hub tokens exchange scene information between the two streams. Camera-aware geometry conditioning and camera-guided noise further support per-view motion control and cross-view geometric exchange. We adapt the model for block-causal generation and distill the 30-step Teacher into a Student with four denoising steps. We train UMI-WM on self-collected open-world and in-studio UMI recordings combined with public manipulation data. To evaluate action following, we use an inverse dynamics model (IDM) to recover relative camera poses and gripper widths from generated videos, and introduce Bimanual Action Fidelity (BAF) to measure how closely these recovered actions match the conditioning actions. UMI-WM achieves the best results on all five reference-video fidelity metrics and the best or competitive performance on VBench and WorldArena. The Teacher and Student outperform all compared baselines in action following, while the four-step Student generates paired wrist videos 6.9 times faster than the Teacher.