Driving · 2026-10-16
What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning
Haitao Shen, Hanyu Liu, Yihan Lin, Huiping Liu, Junbo Cui, Daoheng Li, Yang Li, Jing Zhang, Wenxin Li, Yuan Yao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors build a unified vision-language-action training framework with a sample-level router that assigns each training sample to either action prediction or an auxiliary latent-action (LA) or future-vision (FV) objective. Under the same fixed training budget on LIBERO, CALVIN, and four real-robot tasks, suitable LA/FV supervision outperforms action-only training, even when action prediction receives half or fewer of the training samples.
Why it matters
Robot VLA policies are usually trained by default with action prediction as the dominant supervision. This paper shows that, with explicit supervision allocation, LA and FV prediction can improve policy learning under fixed budgets, and that FV generally works better than LA. This reframes “what to predict” and “how to allocate supervision” as explicit design choices rather than action-dominant defaults.
Method
- Unified VLA framework: a two-stream multimodal diffusion transformer (MMDiT) with separate action and auxiliary experts interacting through joint self-attention.
- Sample-level router under fixed training budget assigns each training sample to either action loss or a chosen auxiliary loss (LA or FV), making supervision allocation explicit.
- Instantiate three LA targets (LAPA and LAPA-DINOv2/LA...v3) and three FV targets (DINOv3, V-JEPA 2, Wan2.2 representations) and evaluate across simulation and real-robot tasks.
Abstract (from arXiv)
Vision-language-action (VLA) policies are typically trained with action prediction as the primary supervision signal, since robots ultimately execute actions. Auxiliary objectives such as latent-action (LA) and future-vision (FV) prediction are often added to improve policy learning, yet they compete with action prediction for the same training budget. We ask whether, under a fixed budget, allocating a substantial share of training samples to such objectives, rather than to further action prediction, can improve policy learning. We build a unified VLA framework that couples a two-stream multimodal diffusion transformer (MMDiT) with a sample-level router, which assigns each training sample to either action prediction or an auxiliary objective and thus makes supervision allocation explicit. Within this framework, we instantiate three LA targets based on LAPA and its DINO-feature variants, and three FV targets based on DINOv3, V-JEPA 2, and Wan2.2 representations. Across LIBERO, CALVIN, and four real-robot tasks, suitable LA and FV supervision outperforms action-only training under the same budget, even when action prediction receives no more than half of the training samples. FV generally outperforms LA, although the strongest LA target remains competitive on CALVIN. Diagnostic analyses suggest that DINO-based LA targets emphasize relative motion, whereas FV targets emphasize absolute position and task identity. We therefore argue that what robot policies learn to predict, and how training is allocated among these targets, should be treated as explicit design choices rather than defaulting to action-dominant training.
Related papers
- ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
- WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
- Forward Dynamics Models
- Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies
- Vision-Predictor-Action: Towards Efficient Embodied AI via Neuro-Symbolic JEPA Predictors
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-15 · Indexed by Pub-AI: 2026-10-16 UTC
What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning
Preprint posted: 2026-10-15 · Original record ↗
- arxiv
- 2610.21915
Explicit source versions (1)
- v1 · 2026-10-15
- Preprint available
Source date: 2026-10-15 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-language-action (VLA) policies are typically trained with action prediction as the primary supervision signal, since robots ultimately execute actions. Auxiliary objectives such as latent-action (LA) and future-vision (FV) prediction are often added to improve policy learning, yet they compete with action prediction for the same training budget. We ask whether, under a fixed budget, allocating a substantial share of training samples to such objectives, rather than to further action prediction, can improve policy learning. We build a unified VLA framework that couples a two-stream multimodal diffusion transformer (MMDiT) with a sample-level router, which assigns each training sample to either action prediction or an auxiliary objective and thus makes supervision allocation explicit. Within this framework, we instantiate three LA targets based on LAPA and its DINO-feature variants, and three FV targets based on DINOv3, V-JEPA 2, and Wan2.2 representations. Across LIBERO, CALVIN, and four real-robot tasks, suitable LA and FV supervision outperforms action-only training under the same budget, even when action prediction receives no more than half of the training samples. FV generally outperforms LA, although the strongest LA target remains competitive on CALVIN. Diagnostic analyses suggest that DINO-based LA targets emphasize relative motion, whereas FV targets emphasize absolute position and task identity. We therefore argue that what robot policies learn to predict, and how training is allocated among these targets, should be treated as explicit design choices rather than defaulting to action-dominant training.