Latent Dynamics · 2026-10-20
Latent World Models Should Plan with Distances Along the Latent Manifold
Zunzhi You, Linwei Tao, Siyu Xu, Yunke Wang, Chang Xu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study LeWM-style latent goal planning that uses Euclidean (L2) distance between predicted terminal and goal latents. They find L2 is reliable only near the goal and forms spurious low-cost basins farther away. They hypothesize latents lie on a curved manifold and propose Neighborhood Graph Cost (NGC) using shortest-path distances on a mutual nearest-neighbor graph over training latents; this changes only the terminal cost (no retraining) and boosts success rates across tasks.
Why it matters
Goal-conditioned latent planning often relies on a simple ambient latent distance. The paper argues that this distance can fail because the learned latents behave like a curved manifold: Euclidean proximity in ambient space does not reliably correspond to proximity in the environment. Their NGC offers a cost-only fix (no retraining) that reshapes the cost landscape by measuring distance along a latent neighborhood graph, improving planning success.
Method
- Analyze LeWM planning cost maps and correlations with physical distances to show L2 ceases to increase away from the goal and creates spurious low-cost basins that trap the planner.
- Propose Neighborhood Graph Cost (NGC): build a mutual k-nearest-neighbor graph over encoded training latents, precompute all-pairs shortest-path distances, and compute a candidate’s terminal cost via shortest-path distance between graph nodes attached to the candidate terminal latent and the goal latent.
- Evaluate NGC vs L2 across four LeWM tasks using the same trained model and CEM planner; also test an ablation that replaces shortest-path distance with Euclidean node distances and an oracle-predictor setup to isolate cost effects.
Abstract (from arXiv)
Latent world models such as LeWM plan by searching for an action sequence whose predicted terminal latent is closest to the goal latent under an ambient norm, such as the Euclidean distance. Planning with such a cost assumes that proximity in the ambient latent space reflects proximity to the goal in the environment. Studying LeWM, we find that this assumption holds only in a small neighborhood around the goal, whereas farther away the L2 cost forms spurious low-cost basins that trap the planner. Yet neighborhood-based embeddings show that the latents retain the underlying physical state. We therefore hypothesize that the latents lie near a curved, low-dimensional manifold, and that the terminal cost should measure the distance along this latent manifold rather than through the ambient space. To estimate this manifold distance, we propose the Neighborhood Graph Cost (NGC), which computes shortest-path distances on a mutual nearest-neighbor graph constructed over training latents. NGC changes only the terminal cost and requires no retraining. This single change improves the planning success rate of LeWM across all four benchmark tasks and nearly solves TwoRoom and Reacher: even at the farthest goal offsets tested, NGC reaches 95% and 100% success, whereas the L2 cost falls to 27% and 81%. These results support the manifold hypothesis and our argument that latent world models should plan with distances along the latent manifold.
Related papers
- How Long, Not How Close: A Learned Temporal Metric for Planning in Latent World Models
- Order-Consistent Latent Geometry for Reliable Short-Horizon Planning in World Models
- ATLAS: Aligned Transport of Latent Structure for Reliable World Model Planning
- Latent Projection for Latent Planning
- Beyond a single latent space: a dual-latent world model for long-horizon planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-18 · Indexed by Pub-AI: 2026-10-20 UTC
Latent World Models Should Plan with Distances Along the Latent Manifold
Preprint posted: 2026-10-18 · Original record ↗
- arxiv
- 2610.25013
Explicit source versions (1)
- v1 · 2026-10-18
- Preprint available
Source date: 2026-10-18 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Latent world models such as LeWM plan by searching for an action sequence whose predicted terminal latent is closest to the goal latent under an ambient norm, such as the Euclidean distance. Planning with such a cost assumes that proximity in the ambient latent space reflects proximity to the goal in the environment. Studying LeWM, we find that this assumption holds only in a small neighborhood around the goal, whereas farther away the L2 cost forms spurious low-cost basins that trap the planner. Yet neighborhood-based embeddings show that the latents retain the underlying physical state. We therefore hypothesize that the latents lie near a curved, low-dimensional manifold, and that the terminal cost should measure the distance along this latent manifold rather than through the ambient space. To estimate this manifold distance, we propose the Neighborhood Graph Cost (NGC), which computes shortest-path distances on a mutual nearest-neighbor graph constructed over training latents. NGC changes only the terminal cost and requires no retraining. This single change improves the planning success rate of LeWM across all four benchmark tasks and nearly solves TwoRoom and Reacher: even at the farthest goal offsets tested, NGC reaches 95% and 100% success, whereas the L2 cost falls to 27% and 81%. These results support the manifold hypothesis and our argument that latent world models should plan with distances along the latent manifold.