Driving · 2026-10-23

DeXplicit: Making Physical State Explicit in Dexterous Manipulation World Models

Adam Lee, Mutian Xu, Yukang Cao, Shulin Tian, Ziwei Liu

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DeXplicit is a dexterous manipulation world model that predicts metric 3D object-centric motion and deformation from an initial scene and a future hand trajectory, then renders video via 3D-guided appearance transport. The authors report moved-point error reductions (27–40% vs PointWorld) and improved object fidelity IoU+ (47% on HOT3D), plus efficient causal generation using two denoising steps per block (57× speedup from a 14B bidirectional configuration).

Why it matters

The authors argue that in dexterous manipulation, action-conditioned video models can produce visually plausible grasps while placing objects at incorrect 3D poses or drifting geometry, because physical state is learned implicitly from pixels. DeXplicit separates physical 3D prediction from video synthesis by using an explicit metric-3D scaffold, aiming to preserve object transport, articulation, and deformation for more faithful interaction forecasting and controllable rendering.

Method

Predict future metric-3D object-centric motion and deformation from initial metric scene points and a future hand trajectory using a lightweight 3D dynamics network with joint-aware action encoding.
Use predicted 3D states to guide video readout: transport initial appearance features through predicted 3D motion and project into future views with a confidence-gated soft edit mask for reliable vs newly visible regions.
Enable efficient causal generation: the authors report two denoising steps per block while conditioning on generated history.

Limitation

DeXplicit assumes an initial metric 3D scene and a future hand trajectory; errors in these inputs may propagate into both physical-state prediction and video synthesis. Appearance transport can only propagate features visible in the initial observation; newly revealed or heavily occluded regions must be synthesized by the renderer, and large disocclusions can still reduce visual fidelity.

Abstract (from arXiv)

Before acting on an object, a robot needs to anticipate how its hand motion will change the physical world and what the resulting scene will look like. Existing approaches typically entangle action-conditioned dynamics, camera motion, and visual synthesis within a video generation model, or represent dynamics as point trajectories without explicit object structure. We introduce DeXplicit, a dexterous manipulation world model that instead predicts how the world evolves explicitly in metric 3D before rendering its visual consequences. Given an initial scene and a future hand trajectory, a lightweight 3D dynamics network predicts object-centric motion and deformation, providing structured and geometrically grounded future world states. These predictions then serve as a strong scaffold for video generation: appearance is transported through the predicted 3D motion and projected into future views, allowing the renderer to focus on completing uncertain and newly visible content rather than implicitly learning world dynamics from pixels. This explicit-3D-first decomposition also enables efficient causal generation, requiring only two denoising steps per block while conditioning on generated history. Across HOT3D, OakInk2, and PhysTwin, DeXplicit reduces moved-point error by 27-40% relative to PointWorld and improves object fidelity IoU+ by 47% on HOT3D under a shared renderer. Our design with two-step causal generation achieves a 57x speedup from our 14B bi-directional configuration in video synthesis.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC

preprint