Latent Dynamics · 2026-10-21

Latent Projection for Latent Planning

Yoo Sung Jang, Michael Ryoo

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose SIGReg-based latent projection for latent-space planning. Because SIGReg-regularized latents are noisy in directions carrying no signal, they fit a constrained normal state-dynamics matrix A and use the induced covariance to project the usual ℓ2 terminal-latent cost onto the k least-uncertain directions. This post hoc change improves planning success, especially on harder 2D navigation settings, with code released.

Why it matters

Latent world models plan by minimizing distance between predicted terminal and goal latents, but the latent contains uncertainty and task-irrelevant content. By exploiting the same regularizer structure to estimate per-direction uncertainty via fitted (normal) dynamics, the method narrows the planning metric to directions that are carried forward with certainty, sharpening action ranking without retraining the world model.

Method

Assume SIGReg makes latent marginals standard Gaussian and model the latent transition as Gauss–Markov: z'|z ~ N(Az, I−AA^T). Fit A from encoded trajectories under constraints (A normal; I−AA^T is a valid covariance).
Use the eigenbasis of Σ=I−AA^T to order latent directions by uncertainty; project both predicted terminal latent and goal latent onto the k most certain directions and compute the projected ℓ2 cost.
Only the cost changes: encoder, predictor, rollout, and planner stay frozen; A is fitted once using least squares (including multi-step fitting).

Limitation

The authors state their method repairs only the cost and cannot compensate for failures of the other two: the information required by the task may be absent from the representation, and the predictor may not advance the latent accurately over the horizon; it also rests on Section 3 assumptions that may not hold under contact-rich dynamics.

Abstract (from arXiv)

Latent world models let an agent plan in latent space. The planner searches for an action sequence that minimizes a cost, and that cost is often the $\ell_2$ distance between the predicted terminal latent and the goal latent. The latent, however, is partly noise: the regularizer that prevents collapse during world model training promotes variance in the directions carrying no signal. We show that the same regularizer also supplies the structure to prune that noise in the latent. SIGReg drives the latent marginal to a standard Gaussian at every step, and assuming further that the latent sequence is a Gauss-Markov process, the transition is linear with its covariance pinned by the map itself, $z' \mid z \sim \mathcal{N}(Az,\, I - AA^\top)$. Fitting the state dynamics $A$ therefore yields the covariance at no additional cost, and in its eigenbasis each latent direction carries its own uncertainty. Furthermore, constraining $A$ to be a normal matrix fixes that basis across timesteps, so one eigenbasis measures the uncertainty over an arbitrary number of steps and serves as a stable basis to compute the planning cost on. We then restrict the $\ell_2$ cost to the $k$-dimensional subspace the state dynamics carry forward with the least uncertainty. The correction is post hoc: the encoder, the predictor, the rollout, and the planner are all left untouched, and only a single constrained matrix, $A$, is fit. We evaluate across six environments and show large gains on the 2D navigation tasks, TwoRoom and PointMaze, raising success rate over the plain $\ell_2$ cost by up to $+54.7$ points under a harder planning setting. Code is available at https://github.com/yjang43/lp2.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-21 UTC

preprint