Driving · 2026-10-02
DeepJEPA: Scaling World Models from Within
Zijian Jin, Yunbei Zhang, Yuanzhe Liu, Ming Liu, Baian Chen, Weirui Ye, Shilong Liu, Marco Pavone
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DeepJEPA treats transition depth as an inner test-time scaling axis for joint-embedding predictive world models. Instead of using a fixed recurrent depth for every imagined transition, it learns when additional recurrent updates are worth computing for each candidate and rollout step during CEM. Across five visual-control settings, DeepJEPA improves or matches the strongest fixed-depth planner while averaging only 1.00–1.26 updates per transition.
Why it matters
The authors argue that scaling world-model planning “outward” (more candidates/horizon/iterations) while using uniform transition computation can waste compute and even degrade planning. DeepJEPA shows gains come from allocating internal computation to decision-critical events (e.g., contact onset and sustained interaction), rather than making all imagined transitions uniformly deeper.
Method
- Introduce DeepJEPA: a weight-tied joint-embedding predictive world model that performs up to Kmax=4 recurrent updates within each imagined transition.
- Train a learned continue head to decide, per candidate–time pair, whether another recurrent update should be computed; inference halts when p_k <= η or at Kmax.
- Integrate with CEM by rolling selected latent predictions forward with candidate budgets, rollout horizon, elites, and CEM iterations kept fixed.
Limitation
Per-task halting thresholds are selected from the measured sweep rather than a held-out validation set; the resulting values characterize the attainable success and compute frontier, not a deployment-time threshold-selection rule. PushT success also remains low in absolute terms. Helped/harmed episode labels depend on both the learned model and the sampled CEM population, so they are not permanent semantic classes.
Abstract (from arXiv)
World-model planners typically scale outward by rolling farther, sampling more trajectories, or optimizing longer, while assigning the same computation to every imagined transition. We show that making every transition uniformly deeper wastes computation and can degrade planning because useful refinement is concentrated at a small set of decision-critical events. We introduce DeepJEPA, a weight-tied joint-embedding predictive world model that treats transition depth as an inner test-time scaling axis and learns when another recurrent update is worth computing for each candidate and rollout step. Across five visual-control settings, DeepJEPA improves or matches the strongest fixed-depth planner while averaging only 1.00-1.26 updates per transition. Its additional computation concentrates at contact onset and sustained object interaction, where latent corrections can change which candidates enter the planner's elite set and which action is selected. Representation probes further show that improved planning does not require uniformly better object-state decodability. DeepJEPA therefore reframes world-model scaling as a problem of allocating internal computation where it can change the planner's decision: think deeper at decision-critical transitions instead of making every rollout uniformly deeper or longer.
Related papers
- The Planning Limits of Latent World Models
- LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
- JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shifts
- H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
- WorldFoundry: Towards Unified Infrastructure for World Intelligence
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC
DeepJEPA: Scaling World Models from Within
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.00368
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World-model planners typically scale outward by rolling farther, sampling more trajectories, or optimizing longer, while assigning the same computation to every imagined transition. We show that making every transition uniformly deeper wastes computation and can degrade planning because useful refinement is concentrated at a small set of decision-critical events. We introduce DeepJEPA, a weight-tied joint-embedding predictive world model that treats transition depth as an inner test-time scaling axis and learns when another recurrent update is worth computing for each candidate and rollout step. Across five visual-control settings, DeepJEPA improves or matches the strongest fixed-depth planner while averaging only 1.00-1.26 updates per transition. Its additional computation concentrates at contact onset and sustained object interaction, where latent corrections can change which candidates enter the planner's elite set and which action is selected. Representation probes further show that improved planning does not require uniformly better object-state decodability. DeepJEPA therefore reframes world-model scaling as a problem of allocating internal computation where it can change the planner's decision: think deeper at decision-critical transitions instead of making every rollout uniformly deeper or longer.
