Latent Dynamics · 2026-11-02

Density-Guided Bootstrap Boundaries for Latent World-Model Reinforcement Learning

Shashank Gupta, Pilhwa Lee

arXiv:2610.41022PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose density-guided bootstrap boundaries for latent world-model RL. Using a score-informed relative density proxy (accumulated negative divergence from a score-corrected auxiliary field), they gate the depth of imagined λ-returns with hard (stop expansion) or soft (shift toward critic) mechanisms. They preserve boundary value estimates and apply survival weights; instantiated in Dreamer V3, they prove strict contraction of the gated Bellman operator and monotonic reduction of conditional target variance.

Why it matters

Fixed imagination horizons treat all trajectory regions equally, even when model prediction quality varies. The method aims to adapt return expansion depth using a trajectory-dependent density proxy, transferring reliance from model rollouts to critic bootstrapping via hard/soft gates. The paper also provides theory (strict contraction; monotonic conditional variance reduction) and clarifies that the proxy is not an exact likelihood under the stochastic world model.

Method

  • Learn an auxiliary feature displacement field and a noise-conditioned score estimator from replay features; form a score-corrected auxiliary field.
  • Estimate accumulated negative divergence along imagined trajectories to build a relative density proxy; use it to produce hard or soft continuation gates for λ-return recursion.
  • Apply survival weights induced by the gates to actor and critic losses; provide Dreamer V3 instantiation and analyze a gated λ-return Bellman operator and conditional target variance.
Abstract (from arXiv)

Latent world models allow reinforcement learning agents to learn policies from imagined trajectories, yet determining the optimal depth of model reliance remains a fundamental challenge. Fixed imagination horizons apply uniform prediction depths regardless of trajectory reliability, exposing target values to compounding model error or over-relying on critic approximations. We introduce density-guided bootstrap boundaries, a trajectory-dependent mechanism that adapts the depth of imagined returns using a score-informed relative density proxy. The proxy accumulates the estimated divergence of a score-corrected auxiliary field formed by a feature displacement network and a noise-conditioned score estimator along imagined paths. We propose two gating mechanisms: a hard boundary that halts return expansion at a discrete state, and a soft boundary that continuously shifts the return recursion toward critic bootstrapping. Both variants preserve boundary value estimates and apply corresponding survival weights to actor and critic losses. Formulated independently of specific dynamics models and instantiated within Dreamer V3, we prove that the resulting gated Bellman operator is a strict contraction mapping and that early bootstrapping monotonically reduces conditional target variance. We clarify the theoretical assumptions distinguishing our expansion proxy from exact path likelihoods. Across continuous-control tasks and Atari Pong, empirical evaluations show that dynamic boundary selection enhances sample efficiency and late-training performance over fixed-horizon and auxiliary-enabled baselines.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-29 · Indexed by Pub-AI: 2026-11-02 UTC

preprint

Density-Guided Bootstrap Boundaries for Latent World-Model Reinforcement Learning

Shashank Gupta, Pilhwa Lee

Preprint posted: 2026-10-29 · Original record ↗

arxiv
2610.41022
Explicit source versions (1)
  • v1 · 2026-10-29
  1. Preprint available

    Source date: 2026-10-29 · First detected: 2026-11-02 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Latent world models allow reinforcement learning agents to learn policies from imagined trajectories, yet determining the optimal depth of model reliance remains a fundamental challenge. Fixed imagination horizons apply uniform prediction depths regardless of trajectory reliability, exposing target values to compounding model error or over-relying on critic approximations. We introduce density-guided bootstrap boundaries, a trajectory-dependent mechanism that adapts the depth of imagined returns using a score-informed relative density proxy. The proxy accumulates the estimated divergence of a score-corrected auxiliary field formed by a feature displacement network and a noise-conditioned score estimator along imagined paths. We propose two gating mechanisms: a hard boundary that halts return expansion at a discrete state, and a soft boundary that continuously shifts the return recursion toward critic bootstrapping. Both variants preserve boundary value estimates and apply corresponding survival weights to actor and critic losses. Formulated independently of specific dynamics models and instantiated within Dreamer V3, we prove that the resulting gated Bellman operator is a strict contraction mapping and that early bootstrapping monotonically reduces conditional target variance. We clarify the theoretical assumptions distinguishing our expansion proxy from exact path likelihoods. Across continuous-control tasks and Atari Pong, empirical evaluations show that dynamic boundary selection enhances sample efficiency and late-training performance over fixed-horizon and auxiliary-enabled baselines.