Driving · 2026-10-28
Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models
Xu Hu, Changjie Wu, Linzhe Shi, Hang Zhang, Chen Yang, Lingjun Zhang, Guowen Zhang, Mu Xu, Junran Peng, Qing Li, Zhaoxiang Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Trace-WAM is a World Action Model that augments video-action learning with future 3D motion prediction as an auxiliary objective. It supervises temporal point correspondence via image-plane displacement and depth change, and uses a three-branch Tri-MoT architecture to jointly learn future video prediction, 3D motion prediction, and action generation. At inference, it follows the Fast-WAM policy path using only current observations.
Why it matters
The authors aim to make WAM representations focus on interaction dynamics for manipulation by adding an auxiliary 3D motion signal, while keeping inference efficient (no future-video or motion prediction required). They report improved few-shot and out-of-distribution performance on LIBERO, LIBERO-Plus, and RoboCasa-GR1, and also show gains in real-world robot tasks.
Method
- Train with joint future video, future 3D motion, and action objectives using a three-branch Mixture-of-Transformer (Tri-MoT) architecture; future video/motion are used only during training.
- Represent motion with grid-anchored trajectories using image-plane displacement plus depth change relative to each anchor’s initial position; predict compact future motion via B-spline coefficients.
- At inference, remove future-video/motion branches and keep the Fast-WAM policy path: single forward pass on current RGB, then action denoising from cached visual features plus language and proprioception.
Abstract (from arXiv)
World Action Models (WAMs) learn representations for robot control by coupling future visual prediction with action generation. Successful manipulation depends primarily on how the robot and task-relevant objects evolve, whereas future-video prediction must also reconstruct the appearance and background of the entire scene. We therefore propose Trace-WAM, which augments video-action learning with future 3D motion prediction as an auxiliary objective. This objective supervises temporal point correspondence together with image-plane displacement and depth change, directly describing the interaction dynamics and spatial changes involved in manipulation. To this end, we adopt a three-branch Mixture-of-Transformer (Tri-MoT) architecture to jointly learn video prediction, motion prediction, and action generation. The motion expert predicts future trajectories from current visual features, allowing motion supervision to shape the representations used for action generation. Because the action expert does not consume predicted trajectories, inference retains the current-observation policy path: a single visual forward pass followed by action denoising. Experiments on RoboCasa-GR1, LIBERO, and LIBERO-Plus demonstrate strong in-distribution performance and out-of-distribution generalization, with particularly clear gains over Fast-WAM under limited demonstrations and distribution shifts.
Related papers
- TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation
- Box-WAM: Box Trajectories as Compact State Representations for World Action Models
- Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction
- De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
- TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC
Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models
Preprint posted: 2026-10-27 · Original record ↗
- arxiv
- 2610.36840
Explicit source versions (1)
- v1 · 2026-10-27
- Preprint available
Source date: 2026-10-27 · First detected: 2026-10-28 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.36840v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World Action Models (WAMs) learn representations for robot control by coupling future visual prediction with action generation. Successful manipulation depends primarily on how the robot and task-relevant objects evolve, whereas future-video prediction must also reconstruct the appearance and background of the entire scene. We therefore propose Trace-WAM, which augments video-action learning with future 3D motion prediction as an auxiliary objective. This objective supervises temporal point correspondence together with image-plane displacement and depth change, directly describing the interaction dynamics and spatial changes involved in manipulation. To this end, we adopt a three-branch Mixture-of-Transformer (Tri-MoT) architecture to jointly learn video prediction, motion prediction, and action generation. The motion expert predicts future trajectories from current visual features, allowing motion supervision to shape the representations used for action generation. Because the action expert does not consume predicted trajectories, inference retains the current-observation policy path: a single visual forward pass followed by action denoising. Experiments on RoboCasa-GR1, LIBERO, and LIBERO-Plus demonstrate strong in-distribution performance and out-of-distribution generalization, with particularly clear gains over Fast-WAM under limited demonstrations and distribution shifts.