Driving · 2026-10-02

DeepJEPA: Scaling World Models from Within

Zijian Jin, Yunbei Zhang, Yuanzhe Liu, Ming Liu, Baian Chen, Weirui Ye, Shilong Liu, Marco Pavone

arXiv:2610.00368PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DeepJEPA treats transition depth as an inner test-time scaling axis for joint-embedding predictive world models. Instead of using a fixed recurrent depth for every imagined transition, it learns when additional recurrent updates are worth computing for each candidate and rollout step during CEM. Across five visual-control settings, DeepJEPA improves or matches the strongest fixed-depth planner while averaging only 1.00–1.26 updates per transition.

Why it matters

The authors argue that scaling world-model planning “outward” (more candidates/horizon/iterations) while using uniform transition computation can waste compute and even degrade planning. DeepJEPA shows gains come from allocating internal computation to decision-critical events (e.g., contact onset and sustained interaction), rather than making all imagined transitions uniformly deeper.

Method

  • Introduce DeepJEPA: a weight-tied joint-embedding predictive world model that performs up to Kmax=4 recurrent updates within each imagined transition.
  • Train a learned continue head to decide, per candidate–time pair, whether another recurrent update should be computed; inference halts when p_k <= η or at Kmax.
  • Integrate with CEM by rolling selected latent predictions forward with candidate budgets, rollout horizon, elites, and CEM iterations kept fixed.

Limitation

Per-task halting thresholds are selected from the measured sweep rather than a held-out validation set; the resulting values characterize the attainable success and compute frontier, not a deployment-time threshold-selection rule. PushT success also remains low in absolute terms. Helped/harmed episode labels depend on both the learned model and the sampled CEM population, so they are not permanent semantic classes.

Abstract (from arXiv)

World-model planners typically scale outward by rolling farther, sampling more trajectories, or optimizing longer, while assigning the same computation to every imagined transition. We show that making every transition uniformly deeper wastes computation and can degrade planning because useful refinement is concentrated at a small set of decision-critical events. We introduce DeepJEPA, a weight-tied joint-embedding predictive world model that treats transition depth as an inner test-time scaling axis and learns when another recurrent update is worth computing for each candidate and rollout step. Across five visual-control settings, DeepJEPA improves or matches the strongest fixed-depth planner while averaging only 1.00-1.26 updates per transition. Its additional computation concentrates at contact onset and sustained object interaction, where latent corrections can change which candidates enter the planner's elite set and which action is selected. Representation probes further show that improved planning does not require uniformly better object-state decodability. DeepJEPA therefore reframes world-model scaling as a problem of allocating internal computation where it can change the planner's decision: think deeper at decision-critical transitions instead of making every rollout uniformly deeper or longer.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC

preprint

DeepJEPA: Scaling World Models from Within

Zijian Jin, Yunbei Zhang, Yuanzhe Liu, Ming Liu, Baian Chen, Weirui Ye, Shilong Liu, Marco Pavone

Preprint posted: 2026-09-30 · Original record ↗

arxiv
2610.00368
Explicit source versions (1)
  • v1 · 2026-09-30
  1. Preprint available

    Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

World-model planners typically scale outward by rolling farther, sampling more trajectories, or optimizing longer, while assigning the same computation to every imagined transition. We show that making every transition uniformly deeper wastes computation and can degrade planning because useful refinement is concentrated at a small set of decision-critical events. We introduce DeepJEPA, a weight-tied joint-embedding predictive world model that treats transition depth as an inner test-time scaling axis and learns when another recurrent update is worth computing for each candidate and rollout step. Across five visual-control settings, DeepJEPA improves or matches the strongest fixed-depth planner while averaging only 1.00-1.26 updates per transition. Its additional computation concentrates at contact onset and sustained object interaction, where latent corrections can change which candidates enter the planner's elite set and which action is selected. Representation probes further show that improved planning does not require uniformly better object-state decodability. DeepJEPA therefore reframes world-model scaling as a problem of allocating internal computation where it can change the planner's decision: think deeper at decision-critical transitions instead of making every rollout uniformly deeper or longer.