Robotics · 2026-10-16
TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control
Menghao Yin, Jiaju Yin, Xiyue Dong, Lixin Xu, Zhenhui Zhang, Ke Zhang, Yating Feng, Renjing Xu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TS-WAM is a lightweight teacher-student vision-language robot control model that learns temporal latent actions and predicts future visual features. It uses vlm-jepa (EMA target branch for future-aware latent-action supervision) coupled with delta-lwm (latent action-conditioned one-step feature-delta prediction) and a flow-matching action head to generate actions from predicted future features. Training is end-to-end on robot demos plus action-free human videos; the EMA teacher is pruned at inference.
Why it matters
The authors target VLA methods that rely on action-labeled demonstrations and lack explicit future visual prediction. TS-WAM makes temporal prediction a central interface and supports learning from action-free videos by using future observations only as training supervision (EMA teacher). The model also couples latent actions to predicted visual dynamics for action generation, aiming to improve generalization and robustness across simulation and real robots.
Method
- Vlm-jepa: online VLM predicts latent actions from current observation+instruction; an EMA teacher VLM reads current+future frames to provide stop-gradient latent-action targets. Teacher is pruned at inference.
- Delta-lwm: a frozen DINOv3 encoder provides feature grids; a trainable decoder predicts one-step feature displacement ΔF conditioned on latent action (with a ranking constraint using batch-mismatched latent).
- Flow-matching action head: a conditional transformer integrates VLM context, robot state/proprioception, and predicted future features to generate an action chunk; integrated vector field at inference.
Limitation
TS-WAM introduces additional training costs: the EMA teacher branch modestly increases GPU memory consumption during training, and conditioning the DiT action expert on predicted future features slows convergence, leading to longer overall training time.
Abstract (from arXiv)
Many vision-language-action policies rely on action-labeled robot demonstrations and lack an explicit mechanism for predicting future visual states. These limitations hinder learning from action-free videos and can encourage overfitting to demonstration conditions, weakening generalization and robustness. We present TS-WAM, a lightweight Teacher-Student World-Action Model that addresses these challenges through self-supervised temporal learning and explicit future prediction. Its vlm-jepa framework learns from heterogeneous data, including robot demonstrations and action-free human videos, using an EMA target branch without requiring a separately trained external teacher. By coupling vlm-jepa with delta-lwm, TS-WAM learns to predict latent actions from the current observation and instruction and uses them to predict future visual features that guide action generation at inference. A unified end-to-end training architecture jointly optimizes temporal representation learning, visual dynamics, and robot control without a separate latent-action or world-model pretraining stage. The EMA target branch is removed at inference, keeping the deployed policy lightweight. Experiments on LIBERO, LIBERO-Plus, and SimplerEnv, together with real-world manipulation experiments on Franka Research 3 and RevoTron, demonstrate that TS-WAM achieves strong manipulation performance, generalization, and robustness.
Related papers
- WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
- Native Action-Prior Learning from Videos for World Action Models
- Rewiring Semantics, Dynamics, and Control: A Simple yet Effective Action-Centric Tri-Stream Transformer
- PaceVLA: Adaptive Temporal Action Quantization in Vision-Language-Action Models
- ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-15 · Indexed by Pub-AI: 2026-10-16 UTC
TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control
Preprint posted: 2026-10-15 · Original record ↗
- arxiv
- 2610.21922
Explicit source versions (1)
- v1 · 2026-10-15
- Preprint available
Source date: 2026-10-15 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Many vision-language-action policies rely on action-labeled robot demonstrations and lack an explicit mechanism for predicting future visual states. These limitations hinder learning from action-free videos and can encourage overfitting to demonstration conditions, weakening generalization and robustness. We present TS-WAM, a lightweight Teacher-Student World-Action Model that addresses these challenges through self-supervised temporal learning and explicit future prediction. Its vlm-jepa framework learns from heterogeneous data, including robot demonstrations and action-free human videos, using an EMA target branch without requiring a separately trained external teacher. By coupling vlm-jepa with delta-lwm, TS-WAM learns to predict latent actions from the current observation and instruction and uses them to predict future visual features that guide action generation at inference. A unified end-to-end training architecture jointly optimizes temporal representation learning, visual dynamics, and robot control without a separate latent-action or world-model pretraining stage. The EMA target branch is removed at inference, keeping the deployed policy lightweight. Experiments on LIBERO, LIBERO-Plus, and SimplerEnv, together with real-world manipulation experiments on Franka Research 3 and RevoTron, demonstrate that TS-WAM achieves strong manipulation performance, generalization, and robustness.