Video World Models · 2026-10-14
Model-Based Online Decision Making via Generative Trajectory Planning
Haldun Balim, Yilun Du, Na Li
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose Generative Trajectory Planning (GTP), an online model-based framework that learns a diffusion-based generative model as a proposal distribution over action sequences. At each timestep, GTP samples candidate action sequences and refines them via beam-search-style iterative diffusion and rollouts through learned dynamics, reward, and value models, using a dynamics ensemble for trajectory-level uncertainty and optimistic exploration.
Why it matters
GTP aims to combine efficient inference with long-horizon reasoning without per-step high-dimensional optimization. It reframes generative models as structured proposal distributions for planning, and uses learned dynamics/reward/value models to score/refine trajectories, while addressing slow value propagation and exploration under model uncertainty.
Method
- Learn a state-conditioned diffusion prior over H-step action sequences; at decision time, sample candidates rather than optimize from scratch.
- Plan via iterative beam search: branch prefixes, forward-diffuse/refine suffixes with diffusion denoising, then prune by scoring rollouts using learned dynamics, reward, and value models.
- Use an ensemble dynamics model to estimate trajectory-level uncertainty and act optimistically (max aggregation), with value learning using median aggregation and inverse-variance horizon weighting.
Abstract (from arXiv)
Online decision-making faces a tension between efficient inference and long-horizon reasoning. Value-based methods rely on short-horizon bootstrapping that limits the propagation of long-term information, while trajectory optimization methods plan explicitly but must solve a high-dimensional optimization problem at every decision step. We propose Generative Trajectory Planning (GTP), a model-based framework that addresses this tension by learning a generative model as a proposal distribution over action sequences. Instead of optimizing sequences from scratch, GTP samples candidate sequences from a diffusion prior and refines them using learned dynamics, reward, and value models, enabling efficient trajectory-level reasoning. Across continuous control and long-horizon manipulation tasks, GTP matches or exceeds state-of-the-art model-free, diffusion-policy, and model-based baselines, and remains stable in regimes where strong baselines degrade. Our results suggest that generative models are best understood not as policies, but as structured proposal distributions for planning, offering a new perspective on how to integrate generative modeling with model-based decision-making.
Related papers
- Dynamics-Informed Action Learning for Goal-Conditioned Latent Planning
- Reasoning as a Distribution: Generative Plan Latents for Vision-Language-Action Models
- Beyond a single latent space: a dual-latent world model for long-horizon planning
- PILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Visual Subgoals and Policy Priors
- TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-14 UTC
Model-Based Online Decision Making via Generative Trajectory Planning
Preprint posted: 2026-10-13 · Original record ↗
- arxiv
- 2610.18677
Explicit source versions (1)
- v1 · 2026-10-13
- Preprint available
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Online decision-making faces a tension between efficient inference and long-horizon reasoning. Value-based methods rely on short-horizon bootstrapping that limits the propagation of long-term information, while trajectory optimization methods plan explicitly but must solve a high-dimensional optimization problem at every decision step. We propose Generative Trajectory Planning (GTP), a model-based framework that addresses this tension by learning a generative model as a proposal distribution over action sequences. Instead of optimizing sequences from scratch, GTP samples candidate sequences from a diffusion prior and refines them using learned dynamics, reward, and value models, enabling efficient trajectory-level reasoning. Across continuous control and long-horizon manipulation tasks, GTP matches or exceeds state-of-the-art model-free, diffusion-policy, and model-based baselines, and remains stable in regimes where strong baselines degrade. Our results suggest that generative models are best understood not as policies, but as structured proposal distributions for planning, offering a new perspective on how to integrate generative modeling with model-based decision-making.
