JEPA · 2026-10-06
Frozen in a Frame: The Velocity Blind Spot in JEPA World Models
Tinghe Zhang, Chunyu Liu, Yu Leon Liu, Zerui Zhao, Jiaheng Chen, Yucheng Xiao, Jiaxing Li, Yunlong Wang, Alex Lamb
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors argue JEPA world models trained from single rendered frames have a structural “velocity blind spot”: without motion blur, a still frame encodes configuration but not instantaneous velocity. They verify on official LeWM checkpoints that linear velocity probes are at chance while position probes reach R^2≈0.94–0.98. They propose RateIdent and TI-JEPA, splitting latent into pose and explicit finite-difference motion, improving stop-at-goal planning and fixing velocity-related diagnostics without privileged supervision.
Why it matters
If a model’s training target cannot identify velocity, then downstream control tasks requiring speed/stop behavior can fail even with powerful encoders. The paper provides a formal, checkable explanation tied to the JEPA single-frame target, establishes diagnostics (RateIdent), and demonstrates a lightweight architectural change (TI-JEPA) that restores velocity information and improves inertial planning results.
Method
- Theory: under instantaneous rendering, any single-frame JEPA target zt cannot identify velocity when velocity is not a measurable function of position; Corollary 1 implies velocity-blind rollouts can’t separate opposite-velocity states.
- Diagnostics (RateIdent): Protocol A linear probe ladder (single/pair/window; plus TI-JEPA’s v head), Protocol B kill experiment using identical initial pixels with opposite velocity, Protocol C stop-at-goal planning requiring velocity.
- Fix (TI-JEPA): split latent into pose code q and explicit finite-difference motion code v, predicted jointly; use a memoryless predictor consuming only (q,v,a).
Limitation
The cross-entropy method used for Protocol C is not a state-of-the-art planner; whether the observed gap persists under stronger planners (e.g., MPPI or a learned policy) is untested. Generalization of Protocol C tests to broader task distributions is also untested.
Abstract (from arXiv)
Joint-embedding predictive architectures (JEPAs) for world modeling train an encoder so a predictor maps a current embedding and action to the next frame's embedding, always from a single rendered frame. This has a structural blind spot: a renderer without motion blur draws a scene from configuration alone, so a single-frame embedding carries no velocity information, for any encoder, including the official released LeWM weights. We confirm this on official checkpoints across four real benchmarks (PushT, Reacher, Cube, TwoRoom): every linear velocity probe sits at or below chance while position probes reach R^2 about 0.95. We introduce RateIdent, a three-stage diagnostic protocol, and TI-JEPA, a lightweight fix splitting the latent into a pose code and an explicit finite-difference motion code, predicted jointly. Across three physically grounded environments, TI-JEPA gives a significant, seed-robust gain on a stop-at-goal planning task over a matched-memory baseline, e.g. 55% lower final distance on Pendulum (p=3.2x10^-10) and 64% on CartPole (p=5.1x10^-15). We reproduce this at official ViT-Tiny plus AdaLN-transformer scale, then push the same recipe onto real dm_control Reacher photographs trained from scratch, where TI-JEPA's branch separation exceeds the memory-having baseline's by roughly 38x, the paper's largest margin. Against a same-footprint recurrent RSSM-style predictor, TI-JEPA matches or beats its rollout accuracy on two of three environments, stays separately probeable for pose and motion, and wins outright on the most coupled one. A checkable formal argument and six evaluated environments show single-frame targets are the wrong object to predict when velocity matters, and a small, interpretable structural change fixes it with no privileged supervision. Code, checkpoints, and the project page are linked below the title.
Related papers
- Hamiltonian JEPA: Action-Conditioned World Models with an Inherited Control State
- LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
- Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- Time-JEPA: Grounding Video World Models with Physical Time Series
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC
Frozen in a Frame: The Velocity Blind Spot in JEPA World Models
Preprint posted: 2026-10-03 · Original record ↗
- arxiv
- 2610.04585
Explicit source versions (1)
- v1 · 2026-10-03
- Preprint available
Source date: 2026-10-03 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Joint-embedding predictive architectures (JEPAs) for world modeling train an encoder so a predictor maps a current embedding and action to the next frame's embedding, always from a single rendered frame. This has a structural blind spot: a renderer without motion blur draws a scene from configuration alone, so a single-frame embedding carries no velocity information, for any encoder, including the official released LeWM weights. We confirm this on official checkpoints across four real benchmarks (PushT, Reacher, Cube, TwoRoom): every linear velocity probe sits at or below chance while position probes reach R^2 about 0.95. We introduce RateIdent, a three-stage diagnostic protocol, and TI-JEPA, a lightweight fix splitting the latent into a pose code and an explicit finite-difference motion code, predicted jointly. Across three physically grounded environments, TI-JEPA gives a significant, seed-robust gain on a stop-at-goal planning task over a matched-memory baseline, e.g. 55% lower final distance on Pendulum (p=3.2x10^-10) and 64% on CartPole (p=5.1x10^-15). We reproduce this at official ViT-Tiny plus AdaLN-transformer scale, then push the same recipe onto real dm_control Reacher photographs trained from scratch, where TI-JEPA's branch separation exceeds the memory-having baseline's by roughly 38x, the paper's largest margin. Against a same-footprint recurrent RSSM-style predictor, TI-JEPA matches or beats its rollout accuracy on two of three environments, stays separately probeable for pose and motion, and wins outright on the most coupled one. A checkable formal argument and six evaluated environments show single-frame targets are the wrong object to predict when velocity matters, and a small, interpretable structural change fixes it with no privileged supervision. Code, checkpoints, and the project page are linked below the title.