Driving · 2026-11-10

WARP-JEPA: World-Model-in-the-Loop Amortized Rollout Planning for Frozen Hierarchical Joint-Embedding Predictive Architectures

Junjie Li, Zhaojiacheng Zhou, Kaixiang Wang, Zihan Wang, QingSong Zhao, Jiong Lou, Jie Li

arXiv:2611.10806PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

WARP-JEPA is an amortized hierarchical planning method for strictly frozen H-JEPA world models. It replaces test-time online gradient descent with learned multi-candidate trajectory proposers (Transformer at the top level, residual MLPs at subordinate two-step bridges). Proposers are trained using model-based rollout costs plus best-of-K behavior cloning, enabling candidate selection without test-time gradient optimization. Across 30 frozen H-JEPA models on four benchmarks, it reports 9.6–18.8× faster inference on long-horizon tasks and up to 33.3 pp improved success.

Why it matters

The authors report that online gradient-based planning dominates episode runtime (96.6–97.7% on long-horizon tasks) and that lower model-predicted rollout cost does not necessarily produce better closed-loop success. WARP-JEPA aims to reduce compute by amortizing planning while keeping world models frozen, and to improve control by pairing rollout-cost objectives with behavior-regularized multi-candidate proposals to reduce exploitation of model errors.

Method

  • Keep all pretrained H-JEPA encoders and predictors strictly frozen; replace per-solve online gradient optimization with learned multi-candidate trajectory proposers.
  • Train each proposer with relaxed WTA rollout-cost objective plus best-of-K behavior cloning, using a world-model-in-the-loop verification at inference for selecting the lowest-cost candidate.
  • Use an asymmetric hierarchy: a lightweight one-layer causal Transformer for top-level macro-action composition and parallel residual MLPs for subordinate two-step bridges; select candidates via frozen-model rollouts (no test-time gradient iterations per solve).
Abstract (from arXiv)

Hierarchical Joint-Embedding Predictive Architectures (H-JEPA) enable long-horizon visual control but rely on costly test-time gradient descent for planning. Our analysis reveals that iterative optimization consumes up to 97.7% of episode runtime and that low predicted rollout costs do not necessarily translate into successful control. We further identify a planning-horizon asymmetry between long-horizon top-level composition and short two-step subordinate planning. To address these challenges, we propose WARP-JEPA, a hierarchical amortized planning framework that replaces online gradient optimization with learned multi-candidate trajectory proposers while keeping the world models frozen. WARP-JEPA combines a lightweight world-model-in-the-loop causal Transformer at the top level with parallel residual MLPs at subordinate levels. The proposers are trained using model-based rollout costs and best-of-K behavior cloning, enabling efficient candidate selection without test-time gradient optimization. Across 30 frozen H-JEPA models on four benchmarks, WARP-JEPA achieves 9.6-18.8x faster episode inference on long-horizon tasks (2.9-3.0x on Cube) and improves task success by up to 33.3 percentage points. These results demonstrate that amortized planning can substantially improve both inference efficiency and control performance without retraining the world models. Code is available at https://github.com/hideinsewer/WARP-JEPA.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

WARP-JEPA: World-Model-in-the-Loop Amortized Rollout Planning for Frozen Hierarchical Joint-Embedding Predictive Architectures

Junjie Li, Zhaojiacheng Zhou, Kaixiang Wang, Zihan Wang, QingSong Zhao, Jiong Lou, Jie Li

First announced by arXiv: 2026-11-10 · Original record ↗

arxiv
2611.10806
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.10806v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Hierarchical Joint-Embedding Predictive Architectures (H-JEPA) enable long-horizon visual control but rely on costly test-time gradient descent for planning. Our analysis reveals that iterative optimization consumes up to 97.7% of episode runtime and that low predicted rollout costs do not necessarily translate into successful control. We further identify a planning-horizon asymmetry between long-horizon top-level composition and short two-step subordinate planning. To address these challenges, we propose WARP-JEPA, a hierarchical amortized planning framework that replaces online gradient optimization with learned multi-candidate trajectory proposers while keeping the world models frozen. WARP-JEPA combines a lightweight world-model-in-the-loop causal Transformer at the top level with parallel residual MLPs at subordinate levels. The proposers are trained using model-based rollout costs and best-of-K behavior cloning, enabling efficient candidate selection without test-time gradient optimization. Across 30 frozen H-JEPA models on four benchmarks, WARP-JEPA achieves 9.6-18.8x faster episode inference on long-horizon tasks (2.9-3.0x on Cube) and improves task success by up to 33.3 percentage points. These results demonstrate that amortized planning can substantially improve both inference efficiency and control performance without retraining the world models. Code is available at https://github.com/hideinsewer/WARP-JEPA.