Robotics · 2026-10-07

DepthWorld: 3D World Model for Robot Manipulation

Jai Bardhan, Josef Sivic, Vladimir Petrik

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DepthWorld targets geometric consistency in robot manipulation world models. The authors build DROID-3D by combining learned stereo depth with a joint factor graph over episodes from the same robot to recover shared kinematic parameters and per-scene extrinsics. Using this calibrated 3D supervision, they train a Stable Video Diffusion model (Stable Video Diffusion backbone with unmodified VAE) that jointly predicts multi-view RGB and metric depth via spatial latent tiling and robot-frame point-map supervision.

Why it matters

The authors argue that RGB-only video world models can produce rollouts that look correct per-frame but lack consistent 3D geometry across views, limiting geometry-dependent robotics uses. Their contributions connect robot-calibrated 3D supervision (DROID-3D) with a world-model architecture that can absorb it without changing the pretrained VAE, and they show depth supervision improves RGB prediction while enabling metric depth for downstream geometric reasoning.

Method

Calibrate teleoperation data into robot-frame 3D: use learned stereo depth plus joint factor graph optimization pooling episodes per robot to recover shared kinematic parameters and per-scene extrinsics (DROID-3D).
Train DepthWorld from a Stable Video Diffusion checkpoint: jointly predict multi-view RGB and depth using spatial latent tiling, keeping the pretrained VAE unchanged.
Add cross-view geometric supervision by projecting predicted depth into the robot base frame with a lightweight point-map head supervised against unified 3D point maps.

Limitation

The authors note that robot-inclusive scoring can dominate metric errors: “DepthWorld receives no credit for rendering the arm,” and points near grasped objects and along the URDF silhouette are dropped due to how the mask is defined.

Abstract (from arXiv)

World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC

preprint