JEPA · 2026-10-07
Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models
Leonardo F. Toso, Yann LeCun, James Anderson, Oumayma Bounou
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors show that standard JEPA training (next-step prediction + anti-collapse like SIGReg) can minimize loss while collapsing controllable unstable modes, preventing stabilization from the learned latent representation. They add an endpoint inverse-dynamics (EP-IDM) action reconstruction objective and prove that exact reconstruction makes the encoder injective on the H-step reachable subspace, preserving unstable modes; they further show convergence to the controllable unstable subspace as H grows.
Why it matters
For control from high-dimensional visual observations, the latent representation must preserve unstable (and marginally stable) directions that feedback needs to correct. This work identifies a failure mode of standard JEPAs (unstable-mode collapse under next-step prediction with anti-collapse) and provides a training modification (inverse dynamics / action reconstruction) with theoretical guarantees tied to detectability and finite-horizon controllability.
Method
- Train a JEPA-style world model with next-step prediction losses (single- or multi-step) plus an anti-collapse regularizer (SIGReg described).
- Show that the combined objective can be minimized while the encoder collapses the unstable subspace, making stabilization impossible from latent states.
- Augment training with an endpoint inverse-dynamics loss (EP-IDM) that reconstructs the full horizon action sequence from initial and final latent states; prove injectivity on the H-step reachable subspace and recovery of the controllable unstable subspace as H grows.
Limitation
The paper establishes theoretical guarantees for linear systems ("We prove our theoretical results for linear systems").
Abstract (from arXiv)
Robotic systems often exhibit unstable modes, along which small perturbations and disturbances can cause unbounded growth unless corrected through feedback. Controlling such systems from high-dimensional visual observations requires representations that preserve these modes. Joint-embedding predictive architectures (JEPAs) provide a natural framework for learning such representations and their dynamics from visual data. However, we demonstrate that next step prediction combined with anti-collapse regularization does not guarantee that controllable unstable modes are preserved: the training loss can be minimized while these modes are collapsed, making stabilization from the learned representation impossible. To address this, we augment world-model training with an action reconstruction objective (i.e., an inverse dynamics loss) that encourages control-aware representations, namely, visual representations that preserve crucial features for control. We prove that exact action reconstruction makes the encoder injective on the finite-horizon reachable subspace. Thus, the encoder cannot discard any state direction reachable by an action sequence within $H$ steps. Moreover, we show that, as $H$ grows, the dominant eigenspace of the finite-horizon controllability Gramian converges to the controllable unstable subspace. We establish our theoretical results for linear systems and demonstrate empirically that our findings extend to nonlinear visual control tasks (CartPole, Walker2D, and PointMaze), highlighting the benefits of control-aware representation learning.
Related papers
- AVL-JEPA: Preventing Causal Dynamics Information Collapse In Joint Embedding Predictive Architecture World Models
- Hamiltonian JEPA: Action-Conditioned World Models with an Inherited Control State
- Bilinear World Models: Learning Representations with Structured Dynamics for Efficient Control
- SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction
- LeWAM: End-to-End World and Action Modeling with JEPAs
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC
Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models
Preprint posted: 2026-10-06 · Original record ↗
- arxiv
- 2610.07540
Explicit source versions (1)
- v1 · 2026-10-06
- Preprint available
Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Robotic systems often exhibit unstable modes, along which small perturbations and disturbances can cause unbounded growth unless corrected through feedback. Controlling such systems from high-dimensional visual observations requires representations that preserve these modes. Joint-embedding predictive architectures (JEPAs) provide a natural framework for learning such representations and their dynamics from visual data. However, we demonstrate that next step prediction combined with anti-collapse regularization does not guarantee that controllable unstable modes are preserved: the training loss can be minimized while these modes are collapsed, making stabilization from the learned representation impossible. To address this, we augment world-model training with an action reconstruction objective (i.e., an inverse dynamics loss) that encourages control-aware representations, namely, visual representations that preserve crucial features for control. We prove that exact action reconstruction makes the encoder injective on the finite-horizon reachable subspace. Thus, the encoder cannot discard any state direction reachable by an action sequence within $H$ steps. Moreover, we show that, as $H$ grows, the dominant eigenspace of the finite-horizon controllability Gramian converges to the controllable unstable subspace. We establish our theoretical results for linear systems and demonstrate empirically that our findings extend to nonlinear visual control tasks (CartPole, Walker2D, and PointMaze), highlighting the benefits of control-aware representation learning.