JEPA · 2026-10-15
One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
Joshua Cheung, Zhang He
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors test whether lower one-step latent prediction error predicts better long-horizon goal-conditioned planning in JEPA world models. Using a frozen-decoder protocol and within-environment checkpoint ladders, they find fidelity improves monotonically while planning saturates after the first epoch, with near-zero correlation (PushT: r = −0.010 over 15 post-epoch-1 checkpoints). Across environments, induced fidelity changes show unstable planning effects.
Why it matters
Checkpoint selection and ablations in latent JEPA world models often rely on one-step prediction loss. The paper provides controlled evidence that this proxy can fail: planning success can flatten even as one-step physical prediction error keeps improving, and the “exchange rate” between fidelity and planning is not stable across tasks. This suggests that one-step fidelity metrics should not be used without downstream planning evaluation and matched controls.
Method
- Introduce a frozen-decoder protocol: fit a decoder from encoder latents to physical simulator state on real observations (world model frozen), then compute one-step error in physical units and decompose it into encoder-representation vs predictor contribution (E_repr, E_pred, E_excess).
- Fine-tune a recent end-to-end JEPA world model for 5 epochs while saving checkpoints each epoch; evaluate goal-conditioned planning using the same checkpoints, with three training seeds and paired planning seeds.
- Assess proxy validity via within-environment saturation (PushT per-epoch ladder), stratification by model-independent training-data support, cross-environment comparisons, and replication under a decoder-free latent drift measure; include a matched continued-training control for an auxiliary objective.
Limitation
Their fidelity measure is one-step and teacher-forced while planning is long-horizon and off-distribution; they state that a rollout-level version is needed to settle generalization. They also evaluate one architecture (15M parameters), three low-dimensional simulated environments, one planner, and warm-started fine-tunes from one checkpoint family, characterizing a neighborhood rather than the space of world models; within-environment correlation is estimated over the 15% fidelity range beyond epoch 1, and planning intervals are limited by three training seeds and 50 planning trajectories.
Abstract (from arXiv)
Latent world models are routinely selected, checkpointed and ablated on one-step prediction loss, on the assumption that a model which predicts the next latent better is a better model. In model-based RL this assumption is known to be unreliable; we ask whether it also fails for latent world models, where the training target is the model's own embedding and the loss is not expressed in interpretable units. We introduce a frozen-decoder protocol reporting one-step error in physical units, decomposed into the part already present in the encoder's representation and the part the predictor adds. Applying it to a recent end-to-end JEPA world model across three continuous-control environments, we move the metric by fine-tuning and measure goal-conditioned planning on the same checkpoints, with three training seeds and paired planning seeds throughout. Within one environment, a per-epoch ladder shows the relationship saturating after a single epoch: the first epoch improves fidelity by 32% and planning by 4.4 pp, but across the fifteen checkpoints that follow -- a further 15% fidelity improvement, monotone in every training seed -- the correlation between one-step error and planning success is r = -0.010. This is not a task-ceiling artifact: stratifying by a model-independent measure of training-data support, planning in the lowest-support stratum starts at 80%, leaving ample headroom, and is nonetheless flat from epoch 1 onward. Across environments the exchange rate is equally unstable: a 93% fidelity reduction on TwoRoom yields -0.7 pp of planning, while a 40% reduction on PushT yields +4.9 pp [+1.1, +8.9]. All results replicate under a decoder-free measure, latent drift. We also show that a matched continued-training control absorbs the entire fidelity gain we had attributed to an auxiliary objective.
Related papers
- When Low Prediction Error Misleads Planning: Diagnosing Representation, Dynamics, and Decision Failures in Latent World Models
- JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shifts
- ATLAS: Aligned Transport of Latent Structure for Reliable World Model Planning
- Action-Consequence Alignment for Reliable Planning and Self-Improving in Latent World Models
- The Planning Limits of Latent World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.19375
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Latent world models are routinely selected, checkpointed and ablated on one-step prediction loss, on the assumption that a model which predicts the next latent better is a better model. In model-based RL this assumption is known to be unreliable; we ask whether it also fails for latent world models, where the training target is the model's own embedding and the loss is not expressed in interpretable units. We introduce a frozen-decoder protocol reporting one-step error in physical units, decomposed into the part already present in the encoder's representation and the part the predictor adds. Applying it to a recent end-to-end JEPA world model across three continuous-control environments, we move the metric by fine-tuning and measure goal-conditioned planning on the same checkpoints, with three training seeds and paired planning seeds throughout. Within one environment, a per-epoch ladder shows the relationship saturating after a single epoch: the first epoch improves fidelity by 32% and planning by 4.4 pp, but across the fifteen checkpoints that follow -- a further 15% fidelity improvement, monotone in every training seed -- the correlation between one-step error and planning success is r = -0.010. This is not a task-ceiling artifact: stratifying by a model-independent measure of training-data support, planning in the lowest-support stratum starts at 80%, leaving ample headroom, and is nonetheless flat from epoch 1 onward. Across environments the exchange rate is equally unstable: a 93% fidelity reduction on TwoRoom yields -0.7 pp of planning, while a 40% reduction on PushT yields +4.9 pp [+1.1, +8.9]. All results replicate under a decoder-free measure, latent drift. We also show that a matched continued-training control absorbs the entire fidelity gain we had attributed to an auxiliary objective.