Latent Dynamics · 2026-10-05
PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
Chunghyun Park, Beomjun Kim, Seungcheol Park, Heeseung Kwon, Yashu Shukla, Seunghoon Sim, Jinwoo Shin, Minsu Cho
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
PointWAM is a 3D world action model for dexterous manipulation. It decomposes the world into scene points and hand keypoints, forecasts both as 3D point trajectories in a shared space-time coordinate frame, and retargets the forecast hand motion to robot actions. Pre-training on 1.15M human demo episodes improves DexJoCo success by 56.9 points; adding scene-trajectory supervision adds 10.9 points.
Why it matters
The authors address limited 3D structure and contact-geometry modeling in existing world action models that use RGB or latent features. By jointly forecasting scene and hands as 3D trajectories and retargeting hand motion to robot actions, PointWAM aims to improve transfer from human demonstrations and to outperform prior methods on dexterous manipulation benchmarks and a real robot.
Method
- Decomposes the world into scene points and hand keypoints and forecasts both as 3D point trajectories in a shared space-time coordinate frame.
- Pre-trains a trajectory forecaster on large-scale human demonstration videos using extracted hand trajectories and scene point trajectories, without requiring task-specific object/keypoint selection.
- Uses an action retargeter to convert forecast hand trajectories (plus robot state) into end-effector pose and finger joint action chunks for robot control.
Limitation
The retargeter learns maps from robot demonstrations and “enforces no explicit joint limits or collision constraints,” because targets are recorded actions that already avoid collisions.
Abstract (from arXiv)
World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.
Related papers
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation
- SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation
- DeXplicit: Making Physical State Explicit in Dexterous Manipulation World Models
- Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
- TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC
PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.02840
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.