Driving · 2026-10-14
World Modeling through Spectral Alignment
Holger Molin, William Peng, Marco Bagatella, Randall Balestriero
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce SpecWM, a JEPA framework that trains a world-model encoder to match a target observation similarity kernel via spectral alignment. Different kernels (physical-state, depth, proprioception, or temporal ordering) determine which quantities the learned representation preserves. They analyze optimality and probing/invariance properties, then show temporal spectral alignment improves CEM planning success over LeWorldModel on OGBench Cube/Scene and CALVIN.
Why it matters
World models must decide what information to retain from complex observations. By explicitly specifying representation geometry through a target similarity kernel, SpecWM provides a way to steer what a latent world model preserves for control, rather than relying only on implicit self-prediction signals.
Method
- SpecWM builds a target similarity kernel from a teacher distance (e.g., Laplacian kernel) and double-centers it, then trains an encoder so its Gram matrix matches the kernel’s (tempered) centered form using a spectral loss.
- The learned embeddings are normalized and used to train an action-conditioned latent predictor; planning uses CEM to score predicted latents by similarity to a goal embedding.
- The framework supports different teachers, including supervised kernels (state, depth, proprioception) and an unsupervised temporal kernel based on observation ordering in trajectories.
Abstract (from arXiv)
How do we tell a world model what information to preserve? We study this question through the lens of spectral representation learning. We introduce SpecWM, a JEPA framework that explicitly specifies desired relationships between observations through a target similarity kernel and trains the encoder to reproduce those relationships in its embedding geometry. Different kernels provide different supervision, ranging from physical state information to a fully self-supervised temporal kernel requiring only the ordering of observations within trajectories. We characterize the global optima of the objective, establish conditions for linear recoverability and invariance to nuisance information, and connect temporal-kernel similarity to the behavioral successor measure under an idealized setting. Empirically, the choice of kernel strongly controls which physical quantities are recoverable from the representation. More importantly, temporal spectral alignment improves CEM planning success over LeWorldModel from 52% to 68% on OGBench Cube, 49% to 77% on OGBench Scene, and 29% to 51% on CALVIN. These results suggest that explicitly specifying representation geometry provides a simple way to steer what latent world models preserve for control.
Related papers
- LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
- ATLAS: Aligned Transport of Latent Structure for Reliable World Model Planning
- Learning Commute-Time-Preserving World Models for Planning
- Counterfactual Action Evaluation, Observation Bottlenecks, and Representation Geometry in Joint-Embedding Predictive World Models
- Time-JEPA: Grounding Video World Models with Physical Time Series
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-11 · Indexed by Pub-AI: 2026-10-14 UTC
World Modeling through Spectral Alignment
Preprint posted: 2026-10-11 · Original record ↗
- arxiv
- 2610.15429
Explicit source versions (1)
- v2 · 2026-10-14
- Explicit source version detected · v2
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Preprint available
Source date: 2026-10-11 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
How do we tell a world model what information to preserve? We study this question through the lens of spectral representation learning. We introduce SpecWM, a JEPA framework that explicitly specifies desired relationships between observations through a target similarity kernel and trains the encoder to reproduce those relationships in its embedding geometry. Different kernels provide different supervision, ranging from physical state information to a fully self-supervised temporal kernel requiring only the ordering of observations within trajectories. We characterize the global optima of the objective, establish conditions for linear recoverability and invariance to nuisance information, and connect temporal-kernel similarity to the behavioral successor measure under an idealized setting. Empirically, the choice of kernel strongly controls which physical quantities are recoverable from the representation. More importantly, temporal spectral alignment improves CEM planning success over LeWorldModel from 52% to 68% on OGBench Cube, 49% to 77% on OGBench Scene, and 29% to 51% on CALVIN. These results suggest that explicitly specifying representation geometry provides a simple way to steer what latent world models preserve for control.