Latent Dynamics · 2026-10-19
Beyond Action Prediction: Learning Search-Free Planning in Latent Space
Guanchen Du, Jinhao Du, Yanpeng Sun, Shan Zhang, Na Zhao, Zechao Li
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose HIJP, a search-free latent-space planning method that replaces test-time action-sequence optimization with a goal-conditioned inverse dynamics model. HIJP trains an inverse model and a latent predictor jointly using hindsight goal relabeling across temporal horizons plus an action-transition consistency loss, so the inverse-to-predictor composition used at test time is explicitly constrained.
Why it matters
HIJP targets a main cost of latent world-model planning: test-time search over action sequences. By training the inverse dynamics and latent prediction models end-to-end (including gradients through goal representations and consistency between predicted actions and latent transitions), the authors aim to enable multi-step receding-horizon planning without online action search, improving both control success and planning efficiency.
Method
- Train a visual encoder fθ, a goal-conditioned inverse dynamics model hξ, and an action-conditioned latent predictor model gϕ end-to-end, using hindsight goals sampled at multiple temporal offsets within trajectory windows.
- Use inverse-loss supervision with gradients routed through both current and goal latent representations to shape goal-side visual features for action prediction.
- Enforce action-transition consistency by feeding predicted actions through gϕ and aligning the resulting next latent with the encoded successor observation; at test time, alternate hξ and gϕ to generate multi-step plans without search.
Limitation
The authors report excluding TwoRoom from ablations because all variants achieve near-100% success rates on this task, making it uninformative for distinguishing different components.
Abstract (from arXiv)
Latent world models learn action-conditioned state transitions in a compact representation space, providing a predictive model for visual planning and control. However, planning with these models typically requires costly test-time optimization over action sequences. We propose to replace this online search with a goal-conditioned inverse model that directly predicts actions from current and goal observations. A key challenge is that action prediction alone neither ensures that goal representations encode action-relevant information nor that predicted actions are consistent with the learned latent prediction model. We introduce Hindsight Inverse JEPA Planning (HIJP), a search-free framework that addresses both aspects during training. HIJP relabels future observations along offline trajectories as goals, providing action supervision across multiple temporal horizons. By retaining gradients through the goal representations, this supervision directly shapes the visual representation. HIJP further feeds predicted actions into the latent predictor model and aligns the resulting next-state representations with encoded successor observations, enforcing action-transition consistency. At inference time, the inverse dynamics model and latent predictor models are applied alternately to generate multi-step action sequences without online search. Across four visual control tasks, HIJP achieves 86.8% overall success and a 1.9-16.5x planning speedup over search-based planning.
Related papers
- H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- Beyond a single latent space: a dual-latent world model for long-horizon planning
- FlexiWorld: Learning and Planning via Flexible Action Chunks Across Multiple Time Scales
- LeWAM: End-to-End World and Action Modeling with JEPAs
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-16 · Indexed by Pub-AI: 2026-10-19 UTC
Beyond Action Prediction: Learning Search-Free Planning in Latent Space
Preprint posted: 2026-10-16 · Original record ↗
- arxiv
- 2610.23615
Explicit source versions (1)
- v1 · 2026-10-16
- Preprint available
Source date: 2026-10-16 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Latent world models learn action-conditioned state transitions in a compact representation space, providing a predictive model for visual planning and control. However, planning with these models typically requires costly test-time optimization over action sequences. We propose to replace this online search with a goal-conditioned inverse model that directly predicts actions from current and goal observations. A key challenge is that action prediction alone neither ensures that goal representations encode action-relevant information nor that predicted actions are consistent with the learned latent prediction model. We introduce Hindsight Inverse JEPA Planning (HIJP), a search-free framework that addresses both aspects during training. HIJP relabels future observations along offline trajectories as goals, providing action supervision across multiple temporal horizons. By retaining gradients through the goal representations, this supervision directly shapes the visual representation. HIJP further feeds predicted actions into the latent predictor model and aligns the resulting next-state representations with encoded successor observations, enforcing action-transition consistency. At inference time, the inverse dynamics model and latent predictor models are applied alternately to generate multi-step action sequences without online search. Across four visual control tasks, HIJP achieves 86.8% overall success and a 1.9-16.5x planning speedup over search-based planning.