Latent Dynamics · 2026-10-01

The Planning Limits of Latent World Models

Ali Alrasheed, Basim Azam, Naveed Akhtar

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors report that latent action-conditioned world models guide action selection reliably only when the goal is within, or slightly beyond, the imagined rollout. With five-step rollouts, targets 5–10 control steps ahead are reliable, while typical tasks require 16–53 steps. The limit persists even with perfect prediction: simulator-based success drops from 92% to 41% when the target moves from 5 to 20 steps ahead. Planning needs longer imagination or closer subgoals.

Why it matters

The paper clarifies when latent world models remain useful for control, focusing on the “plannable range” where imagined futures reliably rank expert actions. By varying predictor capacity, training horizon, rollout horizon, and frozen encoders—and separating learned prediction error from target/rollout mismatch—the authors provide a concrete failure profile and practical guidance for planning design (longer imagined trajectories or subgoals).

Method

Train an action-conditioned predictor over frozen self-supervised visual encoders (V-JEPA 2, V-JEPA 2.1, VideoMAEv2, VideoPrism, DINOv2) and evaluate on Meta-World manipulation and BridgeData V2 offline robot interactions.
Define the plannable range: the largest target lookahead L whose task-averaged expert percentile stays at most 10, using action-ranking against random alternatives from expert demonstrations.
Test factors that could extend the range (predictor capacity, recursive training horizon, planning rollout horizon K, and encoder choice) and run closed-loop MPC/pure imagination, including an exact-dynamics simulator baseline.

Abstract (from arXiv)

World models offer a promising way to help robots understand how the physical world evolves and plan complex behaviours through imagination. Yet existing studies mainly demonstrate what these models can accomplish, leaving unclear when their predictions remain useful for planning and where they fail. We study this question using action-conditioned predictors built on five frozen self-supervised visual backbones: V-JEPA 2, V-JEPA 2.1, VideoMAEv2, VideoPrism, and DINOv2. We use frozen backbones to test representations intended to transfer across environments. We evaluate these models on diverse Meta-World manipulation tasks and real-robot interactions from BridgeData V2. We find that a world model guides action selection reliably only when the goal lies within, or slightly beyond, the trajectory it imagines during planning. With five-step rollouts, the length the predictor was trained on, the world model ranks actions reliably only for targets five to ten control steps ahead, whereas task goals lie 16 to 53 steps away. Neither an 81-fold larger predictor nor longer-rollout training extends this range; the encoder affects both range and closed-loop success, with V-JEPA 2.1 performing most consistently. More fundamentally, the limit persists under perfect prediction: using the real simulator, success falls from 92% to 41% as the target moves from five to twenty steps ahead of a five-step rollout. Planning therefore requires either longer imagined trajectories or closer subgoals. For distant goals, pure imagination succeeds in 23% of episodes, planning with feedback (MPC) raises success to 30%, imagining as far as the goal to 47%, and nearby expert subgoals to 76%. Used within its plannable range, a world model can also improve a vision-language-action (VLA) policy: choosing among eight actions the VLA proposes raises its success from 65% to 77% across 16 different tasks.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC

preprint