Latent Dynamics · 2026-10-21
VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching
Yunzhe Hu, Dong Xu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
VPFlow is an offline goal-reaching method that trains latent world models using closed-form hindsight Monte Carlo (MC) value regression rather than next-state prediction. At test time it uses a value-predictive planner with a one-step flow policy and evolutionary noise-space search guided by generalized advantage (GAE). On OGBench, the authors report higher success rates than model-free baselines and latent planners.
Why it matters
The authors argue that offline planning needs value-predictive representations more than state-predictive ones: imagined rollouts trained to predict goal-reaching value can support planning even when latent next-prediction error is much larger. This reframes world-model learning for offline goal reaching and enables planning that avoids out-of-distribution action queries common in action-space planning.
Method
- Learn latent encoder, latent dynamics, and goal-conditioned value heads jointly using closed-form hindsight Monte Carlo return regression on imagined rollouts (train to predict goal-reaching value, not future states).
- Use a one-step flow policy as a behavior prior; plan by searching in the policy noise space rather than directly in action space.
- Optimize noise candidates using an evolution strategy with generalized advantage estimator (GAE) from imagined rollouts; execute a receding-horizon sequence and replan.
Abstract (from arXiv)
Offline goal-conditioned reinforcement learning enables skill acquisition by reaching any goal from any state without dense rewards. Model-based methods allow for planning by scaling test-time compute to solve complex tasks. Combining both holds appealing promise for pre-training on suboptimal demonstrations and zero-shot adaptation via goal specification and planning. Yet existing methods assume effective planning requires accurate next-state prediction in pixel or latent space. In this work, we show that value prediction is more useful for planning, whereas accurate state prediction is often unnecessary. Our key insight is that offline goal reaching naturally admits a closed-form Monte Carlo return, enabling value regression to replace temporal-difference bootstrapping. We use this return as the sole supervision for latent world models: imagined latents are trained to predict goal-reaching value rather than match future states, implicitly preventing representation collapse. These planning-oriented representations let imperfect world models imagine on the level sets of value field beyond dataset transitions. We then present VPFlow, a value-predictive planner that pairs the world models with a one-step flow policy and plans in its noise space via evolutionary search. It enables policy improvement with imagined generalized advantage and reduces out-of-distribution queries common in action-space planning. On 30 state and pixel OGBench datasets, VPFlow surpasses model-free baselines by at least 30.3 and 10.1 percentage points in success rate, respectively, and outperforms latent planners LeWorldModel, PLDM, and TD-MPC2. It achieves this despite a latent prediction error orders of magnitude larger than with state prediction, while minimizing this error lowers success rate. This demonstrates that value-predictive representations, not exact state prediction, are what offline planning requires.
Related papers
- How Long, Not How Close: A Learned Temporal Metric for Planning in Latent World Models
- The Planning Limits of Latent World Models
- Beyond a single latent space: a dual-latent world model for long-horizon planning
- Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning
- World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-21 UTC
VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching
Preprint posted: 2026-10-19 · Original record ↗
- arxiv
- 2610.27280
Explicit source versions (1)
- v1 · 2026-10-19
- Preprint available
Source date: 2026-10-19 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · OAI arXiv created; not header datestamp (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Offline goal-conditioned reinforcement learning enables skill acquisition by reaching any goal from any state without dense rewards. Model-based methods allow for planning by scaling test-time compute to solve complex tasks. Combining both holds appealing promise for pre-training on suboptimal demonstrations and zero-shot adaptation via goal specification and planning. Yet existing methods assume effective planning requires accurate next-state prediction in pixel or latent space. In this work, we show that value prediction is more useful for planning, whereas accurate state prediction is often unnecessary. Our key insight is that offline goal reaching naturally admits a closed-form Monte Carlo return, enabling value regression to replace temporal-difference bootstrapping. We use this return as the sole supervision for latent world models: imagined latents are trained to predict goal-reaching value rather than match future states, implicitly preventing representation collapse. These planning-oriented representations let imperfect world models imagine on the level sets of value field beyond dataset transitions. We then present VPFlow, a value-predictive planner that pairs the world models with a one-step flow policy and plans in its noise space via evolutionary search. It enables policy improvement with imagined generalized advantage and reduces out-of-distribution queries common in action-space planning. On 30 state and pixel OGBench datasets, VPFlow surpasses model-free baselines by at least 30.3 and 10.1 percentage points in success rate, respectively, and outperforms latent planners LeWorldModel, PLDM, and TD-MPC2. It achieves this despite a latent prediction error orders of magnitude larger than with state prediction, while minimizing this error lowers success rate. This demonstrates that value-predictive representations, not exact state prediction, are what offline planning requires.