Latent Dynamics · 2026-10-02
JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shifts
Zheyuan Zhang, Suyu Ye, Nakul Agarwal, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Daniel Khashabi, Tianmin Shu, Vaishnav Tadiparthi
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
JEPA-TTT adapts the latent dynamics predictor of a pretrained action-conditioned JEPA world model during test time to handle dynamics shifts. Self-supervised updates accumulate across episodes using dense replay, while the visual encoder and reward head stay fixed. It plans with MPC (CEM) without goal images or online environment reward. Across eight dynamics shifts in four continuous-control environments, it improves planning on every shift and reduces autoregressive latent prediction error by 83% on average after 500 episodes.
Why it matters
The authors target a common failure mode of world-model planning: model predictions become unreliable when test-time transition dynamics differ from training. JEPA-TTT shows that persistent self-supervised test-time training of only the latent dynamics predictor—while freezing the representation and task scoring—can improve planning and prediction under dynamics shifts, using only executed actions and consecutive observations.
Method
- Pretrain an action-conditioned JEPA world model (frozen visual encoder and fitted reward head), then adapt only the latent dynamics predictor during deployment via self-supervised latent prediction targets from the frozen encoder.
- Use dense replay: form prediction windows at every valid temporal offset, store them in a growing buffer across episodes, and sample minibatches uniformly for predictor updates.
- Plan with MPC (CEM) by rolling out candidates in latent space with the adapted predictor and scoring with the frozen reward head; no goal image or online environment reward is required.
Limitation
The paper states a testing limitation with PushT: “PushT is the only task without strict CEM success.”
Abstract (from arXiv)
World models enable agents to plan by predicting future states of the environment, but their predictions can become unreliable when test-time dynamics differ from those seen during training. We present JEPA-TTT, which adapts the latent dynamics predictor of a pretrained action-conditioned Joint-Embedding Predictive Architecture world model throughout test time. Self-supervised updates accumulate across episodes, while the visual encoder and reward head remain fixed, preserving the pretrained representation and task objective. Planning requires neither a goal image nor online environment reward. JEPA-TTT uses dense replay, which forms prediction windows at every temporal offset, retains them in a growing buffer, and samples minibatches from that buffer for predictor updates. Across eight dynamics shifts in four continuous-control environments, JEPA-TTT improves planning on every shift. After 500 test-time episodes, it reduces autoregressive latent prediction error by 83% on average and improves planning performance by 153% over the frozen JEPA world model. These results show that persistent self-supervised test-time training can adapt a pretrained latent world model under changed dynamics.
Related papers
- LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
- Enhancing Latent World Models with Associative Memory
- One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
- FlexiWorld: Learning and Planning via Flexible Action Chunks Across Multiple Time Scales
- H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC
JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shifts
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.00722
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World models enable agents to plan by predicting future states of the environment, but their predictions can become unreliable when test-time dynamics differ from those seen during training. We present JEPA-TTT, which adapts the latent dynamics predictor of a pretrained action-conditioned Joint-Embedding Predictive Architecture world model throughout test time. Self-supervised updates accumulate across episodes, while the visual encoder and reward head remain fixed, preserving the pretrained representation and task objective. Planning requires neither a goal image nor online environment reward. JEPA-TTT uses dense replay, which forms prediction windows at every temporal offset, retains them in a growing buffer, and samples minibatches from that buffer for predictor updates. Across eight dynamics shifts in four continuous-control environments, JEPA-TTT improves planning on every shift. After 500 test-time episodes, it reduces autoregressive latent prediction error by 83% on average and improves planning performance by 153% over the frozen JEPA world model. These results show that persistent self-supervised test-time training can adapt a pretrained latent world model under changed dynamics.