Driving · 2026-10-20
TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation
Haohan Geng, Sha Zhang, Weishuo Sun, Guanting Ye, Wenhao Yu, Yuang Xie, Qicheng Zeng, Peng Ye, Jiajun Deng, Jianmin Ji
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TrajWAM is a trajectory-guided world action model for dynamic robotic manipulation. It predicts future target-object masks from a history of object masks, uses a lightweight Trajectory ControlNet to steer a pretrained video model toward trajectory-consistent futures, and conditions an action expert on the resulting future representation. A decoupled one-step distillation enables efficient one-step video-action inference for closed-loop control.
Why it matters
Dynamic manipulation needs anticipation of moving targets plus fast replanning. The authors argue that scene-level video forecasts can underrepresent target-object dynamics and that iterative denoising can introduce latency that makes predictions stale. TrajWAM explicitly grounds future imagination in predicted target motion (mask trajectories) and distills the system for one-step inference, aiming to keep trajectory controllability while reducing latency.
Method
- Use a frozen language-conditioned segmentation model to extract a target-object mask history; train a history-conditioned motion predictor (2D U-Net) to forecast future masks encoding the target’s trajectory.
- Inject the predicted mask trajectory into a pretrained video model via a lightweight Trajectory ControlNet and use asymmetric video–action attention so the action branch conditions on the trajectory-controlled future-video representation.
- Enable one-step deployment with decoupled distillation: video backbone distillation, Trajectory ControlNet realignment, and action rollout distillation to adapt the action expert to the compressed one-step video representation.
Abstract (from arXiv)
Dynamic object manipulation requires robots to anticipate target motion while replanning fast enough to avoid executing outdated actions. Existing approaches either rely on fast reactive control without an explicit future target state or encode anticipated motion only implicitly. World Action Models (WAMs) offer a promising alternative by coupling future visual prediction with action generation, but scene-level video objectives may underrepresent target dynamics that are critical for manipulation, while iterative video-action denoising introduces substantial latency. We present TrajWAM, a trajectory-guided world action model for efficient dynamic robotic manipulation. Our key insight is to explicitly ground future visual imagination in predicted target-object motion and use the resulting future representation to condition action generation, while preserving this anticipation pathway after acceleration. Given a short history of object masks, a history-conditioned motion predictor forecasts future masks encoding the target object's trajectory. A lightweight Trajectory ControlNet then steers a pretrained video model toward trajectory-consistent futures. The resulting future representation conditions an action expert to predict executable action chunks. To enable one-step inference without breaking this interface, we introduce a decoupled distillation scheme that combines distribution matching for the video backbone, Trajectory ControlNet realignment, and action rollout distillation to support accurate one-step action prediction conditioned on the compressed future representation. Comprehensive evaluations in simulation and the real world show that TrajWAM achieves state-of-the-art performance on dynamic object manipulation tasks while enabling efficient one-step video-action inference. Project page: https://kevingeng3.github.io/TrajWAM/.
Related papers
- Box-WAM: Box Trajectories as Compact State Representations for World Action Models
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
- De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
- Dynamic Manipulation with World-Action Models via Counterfactual Planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-20 UTC
TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation
Preprint posted: 2026-10-19 · Original record ↗
- arxiv
- 2610.25763
Explicit source versions (1)
- v1 · 2026-10-19
- Preprint available
Source date: 2026-10-19 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Dynamic object manipulation requires robots to anticipate target motion while replanning fast enough to avoid executing outdated actions. Existing approaches either rely on fast reactive control without an explicit future target state or encode anticipated motion only implicitly. World Action Models (WAMs) offer a promising alternative by coupling future visual prediction with action generation, but scene-level video objectives may underrepresent target dynamics that are critical for manipulation, while iterative video-action denoising introduces substantial latency. We present TrajWAM, a trajectory-guided world action model for efficient dynamic robotic manipulation. Our key insight is to explicitly ground future visual imagination in predicted target-object motion and use the resulting future representation to condition action generation, while preserving this anticipation pathway after acceleration. Given a short history of object masks, a history-conditioned motion predictor forecasts future masks encoding the target object's trajectory. A lightweight Trajectory ControlNet then steers a pretrained video model toward trajectory-consistent futures. The resulting future representation conditions an action expert to predict executable action chunks. To enable one-step inference without breaking this interface, we introduce a decoupled distillation scheme that combines distribution matching for the video backbone, Trajectory ControlNet realignment, and action rollout distillation to support accurate one-step action prediction conditioned on the compressed future representation. Comprehensive evaluations in simulation and the real world show that TrajWAM achieves state-of-the-art performance on dynamic object manipulation tasks while enabling efficient one-step video-action inference. Project page: https://kevingeng3.github.io/TrajWAM/.