JEPA · 2026-10-05
Keeping JEPA World Models Plannable When Little of the Frame Moves
Florian Strohm, Patrick Wagner, Jannik Schwab, Marco Huber
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors build SLIM, a pushing benchmark with paired visual and language goals on identical scenes where little of the frame responds to actions. They find a vanilla JEPA latent world model (LeWM) largely fails (0.003 success) because its encoder latent is nearly action-insensitive. An inverse-dynamics auxiliary loss (LeWM+I) repairs action-relevant latent dynamics and enables language-goal planning via a small head.
Why it matters
The work targets a specific failure mode for latent world models used for planning: when actions barely move much of the frame, the encoder can become action-insensitive, making the latent cost landscape flat. By diagnosing with probes and repairing with an inverse-dynamics auxiliary loss, the authors enable a language-goal interface (language instructions instead of goal frames) to plan and generalize to unseen nouns within their SLIM setup.
Method
- Build SLIM (small-object, language-instructed, multi-object pushing) with paired visual and language goals on identical scenes and controlled “response-to-actions” regimes.
- Diagnose vanilla LeWM failure via probes (action sensitivity, decodability, rollout error, flat CEM cost) and test a response-to-actions sweep.
- Repair LeWM with a shared inverse-dynamics auxiliary loss on consecutive encoder latents and predicted latents (encoder-side term via backprop; predictor-side via stop-gradient), then use a small language-goal head to plan without retraining the world model.
Abstract (from arXiv)
Specifying a goal in language rather than as a goal frame is a natural interface for planning with a latent world model, but testing it needs scenes in which language must discriminate between several objects. We build SLIM, a pushing benchmark with several small objects and paired visual and language goals on identical scenes. On SLIM a LeWM world model that solves PushT succeeds on under 1% of trials, although a scripted controller with simulator state solves every tier. Probes locate the failure in the encoder: its latent is nearly action-insensitive, neither pusher nor object positions can be decoded from it, and rollouts are no better than copying the current latent forward. One inverse-dynamics auxiliary loss, applied to encoder latents and to predicted latents through a shared head discarded at test time, restores every probe and raises success from 0.003 to 0.35 (0.16 on the hard pushing tier, where a goal-agnostic policy scores zero), and improves PushT at twice the trained horizon. Controls attribute the repair to the gradient into the encoder, and a response sweep shows that the vanilla model plans once enough of the frame responds to actions. A cheap action-sensitivity probe, computable without environment access, acts as an empirical necessary condition: all configurations below its threshold failed to plan. On the repaired latent, a small language-goal head plans from sentences without retraining the world model: it reaches 0.84 on navigation (visual-goal oracle 1.00), follows the named zone when it is swapped with a decoy, and degrades gracefully to unseen nouns. A single goal sentence rarely completes a push, but given the push as a sequence of stage sentences the head raises success on the medium and hard pushing tiers from 0.04 to 0.25, on par with the goal-frame oracle, also when the switch between stages is read from the latent alone.
Related papers
- The Planning Limits of Latent World Models
- Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- LaLa-World: Scaling Latent World Models with Natural Language Actions
- One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC
Keeping JEPA World Models Plannable When Little of the Frame Moves
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.03137
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Specifying a goal in language rather than as a goal frame is a natural interface for planning with a latent world model, but testing it needs scenes in which language must discriminate between several objects. We build SLIM, a pushing benchmark with several small objects and paired visual and language goals on identical scenes. On SLIM a LeWM world model that solves PushT succeeds on under 1% of trials, although a scripted controller with simulator state solves every tier. Probes locate the failure in the encoder: its latent is nearly action-insensitive, neither pusher nor object positions can be decoded from it, and rollouts are no better than copying the current latent forward. One inverse-dynamics auxiliary loss, applied to encoder latents and to predicted latents through a shared head discarded at test time, restores every probe and raises success from 0.003 to 0.35 (0.16 on the hard pushing tier, where a goal-agnostic policy scores zero), and improves PushT at twice the trained horizon. Controls attribute the repair to the gradient into the encoder, and a response sweep shows that the vanilla model plans once enough of the frame responds to actions. A cheap action-sensitivity probe, computable without environment access, acts as an empirical necessary condition: all configurations below its threshold failed to plan. On the repaired latent, a small language-goal head plans from sentences without retraining the world model: it reaches 0.84 on navigation (visual-goal oracle 1.00), follows the named zone when it is swapped with a decoy, and degrades gracefully to unseen nouns. A single goal sentence rarely completes a push, but given the push as a sequence of stage sentences the head raises success on the medium and hard pushing tiers from 0.04 to 0.25, on par with the goal-frame oracle, also when the switch between stages is read from the latent alone.