Driving · 2026-10-01

Learning to Plan from Random Exploration

Deqian Kong, Guangyan Sun, Sheng Cheng, Sirui Xie, Bo Pang, Jianwen Xie, Tony Geng, Caiwen Ding, Ying Nian Wu

arXiv:2609.38383PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors learn multi-horizon temporal relations from random exploration without action or reward labels, using a conditional energy-based model trained with noise-contrastive estimation on observation pairs. A separate one-step local dynamics model predicts candidate action outcomes, and a planner scores progress toward a goal across horizons using learned temporal log-density ratios. They demonstrate long-range planning in mazes from both states and images, and further egocentric navigation and manipulation planning.

Why it matters

The paper investigates whether representations from random (goal-agnostic) trajectories can support long-range planning without policy-improvement training. The approach separates long-range guidance (multi-horizon temporal relations) from local action outcome prediction, enabling closed-loop replanning with both models fixed and providing multiscale “cognitive map” structure consistent with a random-walk analysis.

Method

  • Learn horizon-conditioned temporal log-density ratios with a conditional energy-based model using only observation pairs, trained by noise-contrastive estimation (no action/reward labels).
  • Train a separate local dynamics model for one-step action outcomes (action-labeled transitions only at horizon τ=1).
  • Plan in closed loop: predict candidate action outcomes with the local model, then score progress toward the goal using temporal relations at multiple horizons via greedy (β→0+) or potential-ascent (β→∞) objectives; execute one action then replan with models fixed.
Abstract (from arXiv)

Random exploration reveals how an environment can be traversed before a goal is specified. Can this experience support long-range planning without policy-improvement training? Our random-walk analysis explains what temporal relations contain: short horizons reveal geodesic geometry in the diffusion limit, while longer horizons reveal connectivity between regions before mixing removes these distinctions. We learn these relations with a conditional energy-based model that estimates temporal log-density ratios through horizon-conditioned embeddings. The model is trained on observation pairs by noise-contrastive estimation, without action or reward labels. The planner queries these learned relations at different horizons as it moves toward the goal. At test time, a separate local dynamics model predicts candidate action outcomes, and the temporal model evaluates their progress toward the goal by selecting or aggregating estimated improvements across horizons. The agent executes one action and replans with both models fixed. Experiments demonstrate long-range maze planning from random exploration using states and images. Learned score fields, embedding probes, and planned routes exhibit properties of a multiscale cognitive map. We further demonstrate egocentric navigation from random exploration and manipulation planning from suboptimal data.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-29 · Indexed by Pub-AI: 2026-10-01 UTC

preprint

Learning to Plan from Random Exploration

Deqian Kong, Guangyan Sun, Sheng Cheng, Sirui Xie, Bo Pang, Jianwen Xie, Tony Geng, Caiwen Ding, Ying Nian Wu

Preprint posted: 2026-09-29 · Original record ↗

arxiv
2609.38383
Explicit source versions (1)
  • v1 · 2026-09-29
  1. Preprint available

    Source date: 2026-09-29 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Random exploration reveals how an environment can be traversed before a goal is specified. Can this experience support long-range planning without policy-improvement training? Our random-walk analysis explains what temporal relations contain: short horizons reveal geodesic geometry in the diffusion limit, while longer horizons reveal connectivity between regions before mixing removes these distinctions. We learn these relations with a conditional energy-based model that estimates temporal log-density ratios through horizon-conditioned embeddings. The model is trained on observation pairs by noise-contrastive estimation, without action or reward labels. The planner queries these learned relations at different horizons as it moves toward the goal. At test time, a separate local dynamics model predicts candidate action outcomes, and the temporal model evaluates their progress toward the goal by selecting or aggregating estimated improvements across horizons. The agent executes one action and replans with both models fixed. Experiments demonstrate long-range maze planning from random exploration using states and images. Learned score fields, embedding probes, and planned routes exhibit properties of a multiscale cognitive map. We further demonstrate egocentric navigation from random exploration and manipulation planning from suboptimal data.