Driving · 2026-11-10
DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight
Zhongxi Chen, Yi Yang, Xinyi Chen, Mingzhe Ye, Yaping Li, Zhihong Liu, Congsheng Xu, Haonan Wang, Zhijie Deng
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DexForesight trains an action-conditioned spatial world model from offline scene depth, hand geometry, and demonstration trajectories to implicitly learn spatial foresight (3D hand-point motion and visual feature changes). The world model is frozen and its predictive query representations are aligned to a dexterous policy’s latents via query alignment, enabling deployment with only RGB observations, language, and robot state (no depth input or explicit future prediction).
Why it matters
Dexterous manipulation needs spatial foresight over articulated fingers (including occluded parts) and changing scene geometry, but explicit future decoding at deployment is computationally costly. DexForesight transfers predictive hand–scene dynamics from a frozen world model into policy latent features, aiming to retain control-relevant foresight without world-model execution or future decoding.
Method
- Train an action-conditioned spatial world model using offline scene depth (real robots via StereoAnyVideo; simulation rendered), kinematic hand geometry from URDF-linked forward kinematics, and demonstration controls to predict 3D hand-point motion and visual feature changes.
- Freeze the world model and align policy queries with the world model’s predictive query features (query alignment), conditioning the action expert with aligned latent features.
- Deploy only the policy: it uses RGB observations, language, and robot state, with the world model removed and no online depth reconstruction or explicit future generation.
Limitation
The authors state that architectural and training differences preclude treating either WLA or Fast-WAM as a one-variable causal control in the real-robot comparison.
Abstract (from arXiv)
Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline. Project page https://dexforesight.github.io/
Related papers
- PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
- DeXplicit: Making Physical State Explicit in Dexterous Manipulation World Models
- Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
- ForceLDM: A Force-aware Latent Dynamics Model for Contact-Rich Manipulation
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.11166
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11166v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline.Project page https://dexforesight.github.io/
