Latent Dynamics · 2026-10-09

PLaW-VLA: Predictive Latent World Modeling for Vision-Language-Action Policies

Yu Liu, Hetian Guo, Tianlv Huang, Ziyi Cai, Wudi Chen, Hantang Wang, Qiutong Liu, Yingzhi Peng, Wei Han, Peijun Tang, Jianan Wang, Zipei Fan, Zhiyuan Zha, Xuan Song

arXiv:2610.12285 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

PLaW-VLA predicts task-relevant future latent states in a pretrained V-JEPA 2 representation space and conditions a VLA action model on these predicted futures using structured causal attention. Built on a Mixture-of-Transformers with three causally ordered experts, it improves long-horizon control and distribution-shift generalization (e.g., +11.8 pp on RoboTwin Hard Horizon III; +1.77 pp on zero-shot LIBERO-Plus) with parallel latent prediction and ~1/19 the inference latency of generative world–action modeling at comparable performance.

Why it matters

For long-horizon vision-language-action control under partial observability, the authors argue that future modeling should focus on control-relevant state transitions rather than reconstructing low-level visual details. PLaW-VLA’s prediction-oriented latent target and explicit conditioning via predicted future states aim to reduce future-prediction burden while improving robustness under distribution shift and maintaining strong inference efficiency compared to generative world–action approaches.

Method

Uses a Mixture-of-Transformers with three causally ordered experts: vision-language semantic grounding (PaliGemma), latent world modeling (predict future latents in a frozen V-JEPA 2 space), and continuous action generation conditioned on grounded semantics, history latents, and predicted future latents.
Predicts future task-relevant latent states using learnable future queries; structured causal cross-expert attention preserves information flow from semantics → latent prediction → action decoding and prevents future leakage.
Training: Stage I pretrains LWM on action-free videos; Stage II jointly trains future-state prediction and action generation on robot trajectories; Stage III adapts to target tasks/embodiments; world-model dropout stochastically skips predictive conditioning during training updates.

Abstract (from arXiv)

Learning to predict how the world evolves can provide vision-language-action (VLA) policies with predictive context for long-horizon control, but its effectiveness depends on what future representation is modeled and how it conditions action generation. We introduce PLaW-VLA, which models task-relevant future states in a pretrained prediction-oriented representation space, reducing the need to predict control-irrelevant visual details. Built on a Mixture-of-Transformers architecture, PLaW-VLA conditions action generation on observation history, current task semantics, and predicted future states through structured causal attention. Experiments show a +11.8 percentage-point (pp) gain over reactive policies on RoboTwin Hard Horizon III and a +1.77 pp gain over reconstruction-oriented latent prediction on zero-shot LIBERO-Plus, supporting improved long-horizon control and generalization under distribution shift, respectively. By avoiding low-level visual reconstruction, PLaW-VLA lowers the burden of future prediction, enabling a lightweight latent world model with parallel future prediction and about 1/19 the inference latency of generative world-action modeling at comparable policy performance.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-09 UTC

preprint