Robotics · 2026-10-07
ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
Yuan Xu, Yixiang Chen, Qisen Ma, Jiabing Yang, Peiyan Li, Kai Wang, Jianhua Yang, Jianlou Si, Jun Huang, Jing Liu, Nianfeng Liu, Yan Huang, Liang Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ViDAL introduces a Visual Dynamics-grounded Action Latent Space for Vision-Language-Action (VLA) models. It trains an Action VAE to reconstruct action chunks while aligning action latents with future visual dynamics extracted by a frozen video VAE. The frozen Action VAE is then a plug-in action interface for downstream VLA policies and enables optional future-video prediction. The authors report improved success on LIBERO, RoboTwin 2.0, and two real-robot platforms.
Why it matters
VLA action representations are often optimized for action reconstruction, but ViDAL explicitly grounds continuous action latents in how actions change future visual dynamics. The authors’ plug-in Action VAE design claims to work across different VLA action experts (regression and flow-matching) without modifying the VLM backbone, aiming to improve long-horizon, multi-task manipulation and transfer to real robots.
Method
- Train an Action VAE (with reconstruction + KL + DCT-domain loss) whose action latent is supervised to predict future visual latents from a frozen video VAE via a lightweight visual-dynamics predictor.
- Deployment: freeze only the Action VAE as a plug-in action representation; the VLA policy predicts an encoded action latent and the Action VAE decoder maps it back to executable action chunks (no video model needed).
- Optionally decode predicted future visual latents for future-video prediction as qualitative capability.
Abstract (from arXiv)
Vision-Language-Action (VLA) models have become a central paradigm for robot policy learning, which predict actions in three forms: raw action chunks, discrete action tokens, or continuous action latents. However, existing action representations primarily model action trajectories, with limited consideration of the visual dynamics induced by these actions. We introduce ViDAL, a Visual Dynamics-grounded Action Latent Space that anchors continuous action latents in the future visual dynamics of the scene. Specifically, ViDAL learns action latent space by training an Action Variational Autoencoder (Action VAE) to reconstruct action chunks while aligning its latent with future scene dynamics. When integrated into downstream robot policies, the proposed Action VAE serves as a plug-in action interface compatible with multiple VLA architectures and enables optional future-video prediction as an additional capability. Empirically, ViDAL outperforms competitive baselines on LIBERO with 98.1% average success, improves a multi-task $\pi_{0.5}$ policy on RoboTwin 2.0 from 54.3% to 65.5% (Clean) and from 33.2% to 43.1% (Random) success rates over 50 dual-arm tasks, and yields 20.0% and 23.4% absolute success-rate gains on real-world single-arm Franka and dual-arm ARX robot platforms.
Related papers
- What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning
- WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
- TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control
- Juno: Taming Predictive Latents for Vision-Language-Action Models
- Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC
ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
Preprint posted: 2026-10-06 · Original record ↗
- arxiv
- 2610.08150
Explicit source versions (1)
- v1 · 2026-10-06
- Preprint available
Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-Language-Action (VLA) models have become a central paradigm for robot policy learning, which predict actions in three forms: raw action chunks, discrete action tokens, or continuous action latents. However, existing action representations primarily model action trajectories, with limited consideration of the visual dynamics induced by these actions. We introduce ViDAL, a Visual Dynamics-grounded Action Latent Space that anchors continuous action latents in the future visual dynamics of the scene. Specifically, ViDAL learns action latent space by training an Action Variational Autoencoder (Action VAE) to reconstruct action chunks while aligning its latent with future scene dynamics. When integrated into downstream robot policies, the proposed Action VAE serves as a plug-in action interface compatible with multiple VLA architectures and enables optional future-video prediction as an additional capability. Empirically, ViDAL outperforms competitive baselines on LIBERO with 98.1% average success, improves a multi-task $π_{0.5}$ policy on RoboTwin 2.0 from 54.3% to 65.5% (Clean) and from 33.2% to 43.1% (Random) success rates over 50 dual-arm tasks, and yields 20.0% and 23.4% absolute success-rate gains on real-world single-arm Franka and dual-arm ARX robot platforms.