Robotics · 2026-11-06
Spatial-UMI-Dream: Scaling World Action Models with Metric Geometry for UMI Manipulation
Weiduo Yuan, Ze Yuan, Sihan Liu, Jionghao Wang, Liang Heng, Zhenyu Zhao, Yue Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Spatial-UMI-Dream is a 5B spatial World Action Model pretrained on 605 hours of RGB-D UMI data with two-stage training (world-only adaptation, then joint world-action learning). It jointly predicts future RGB, fisheye-aligned metric depth, and actions using stereo-derived metric depth aligned to moving wrist cameras. It is evaluated on RoboTwin-UMI (20 tasks) under environment and embodiment shifts, and on six real-world tasks.
Why it matters
The authors target UMI’s moving, close-range gripper cameras where metric geometry is implicit in RGB. By explicitly incorporating aligned metric depth into a spatial WAM and evaluating with a benchmark that separates environment vs embodiment changes, the work tests whether robot-independent UMI data yields robust cross-embodiment manipulation and reliable control under appearance shifts.
Method
- Two-stage training: world-only adaptation predicts future RGB-D canvases from UMI RGB-D history and language; joint world-action training adds action generation with robot-state and minimizes L_world + λa L_action.
- Metric RGB-D: stereo cameras on each gripper estimate disparity and metric depth (Z-depth) and align it to the native fisheye RGB; depth is encoded via a pretrained RGB VAE and placed with the two grippers’ RGB in a unified dual-gripper RGB-D canvas for video-token modeling.
- RoboTwin-UMI evaluation: a 20-task simulation benchmark with UMI wrist-mounted fisheye observations; environment robustness and cross-embodiment transfer are tested separately and jointly without additional fine-tuning.
Abstract (from arXiv)
The Universal Manipulation Interface (UMI) enables scalable demonstration collection independently of a specific robot. However, its moving, close-range gripper cameras entangle camera and object motion without a stable external view, leaving the metric geometry needed for precise manipulation implicit in RGB. We introduce Spatial-UMI-Dream, a 5B spatial World Action Model (WAM) pretrained on 605 hours of RGB-D UMI data through two-stage training -- world-only adaptation followed by joint world-action learning -- to jointly predict future RGB, fisheye-aligned metric depth, and actions. Stereo-derived depth makes scene scale and spatial relationships explicit within a unified dual-gripper RGB-D representation. Furthermore, evaluating UMI policies requires testing whether UMI's robot-independent demonstration collection translates into policy generalization across robot embodiments, alongside robustness to environmental changes. We introduce RoboTwin-UMI, a twenty-task simulation benchmark built on RoboTwin with UMI wrist-mounted fisheye observations, enabling controlled, repeatable evaluation of UMI policies under environment and embodiment changes separately and jointly without additional fine-tuning. On RoboTwin-UMI, Spatial-UMI-Dream achieves 79.10% in-domain success and 72.15% under combined embodiment and environment shifts, exceeding the strongest evaluated baseline in each setting by 4.65 and 10.30 percentage points, respectively. It also exhibits the smallest success-rate drop under embodiment transfer among the evaluated policies. In a matched pretraining ablation, RGB-only pretraining leads to a 5.75-percentage-point decrease in in-domain success compared with RGB-D pretraining. Across six real-world tasks, Spatial-UMI-Dream achieves 85.0% average success, exceeding the strongest baseline by 24.2 percentage points. Project page: https://weiduoyuan.com/spatial-umi-dream/
Related papers
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation
- UMI-WM: Interactive World Modeling for Bimanual Manipulation from Wrist Cameras
- XGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation
- Physical Canvas: A Vision Tactile World Model For Robot Policy Evaluation And Learning
- DepthWorld: 3D World Model for Robot Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-06 · Indexed by Pub-AI: 2026-11-06 UTC
Spatial-UMI-Dream: Scaling World Action Models with Metric Geometry for UMI Manipulation
First announced by arXiv: 2026-11-06 · Original record ↗
- arxiv
- 2611.07229
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-06 · First detected: 2026-11-06 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.07229v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
The Universal Manipulation Interface (UMI) enables scalable demonstration collection independently of a specific robot. However, its moving, close-range gripper cameras entangle camera and object motion without a stable external view, leaving the metric geometry needed for precise manipulation implicit in RGB. We introduce Spatial-UMI-Dream, a 5B spatial World Action Model (WAM) pretrained on 605 hours of RGB-D UMI data through two-stage training -- world-only adaptation followed by joint world-action learning -- to jointly predict future RGB, fisheye-aligned metric depth, and actions. Stereo-derived depth makes scene scale and spatial relationships explicit within a unified dual-gripper RGB-D representation. Furthermore, evaluating UMI policies requires testing whether UMI's robot-independent demonstration collection translates into policy generalization across robot embodiments, alongside robustness to environmental changes. We introduce RoboTwin-UMI, a twenty-task simulation benchmark built on RoboTwin with UMI wrist-mounted fisheye observations, enabling controlled, repeatable evaluation of UMI policies under environment and embodiment changes separately and jointly without additional fine-tuning. On RoboTwin-UMI, Spatial-UMI-Dream achieves 79.10% in-domain success and 72.15% under combined embodiment and environment shifts, exceeding the strongest evaluated baseline in each setting by 4.65 and 10.30 percentage points, respectively. It also exhibits the smallest success-rate drop under embodiment transfer among the evaluated policies. In a matched pretraining ablation, RGB-only pretraining leads to a 5.75-percentage-point decrease in in-domain success compared with RGB-D pretraining. Across six real-world tasks, Spatial-UMI-Dream achieves 85.0% average success, exceeding the strongest baseline by 24.2 percentage points. Project page: https://weiduoyuan.com/spatial-umi-dream/
