Latent Dynamics · 2026-11-06

La-JEPA: Compact Robot Policies with Latent World Modeling

Mateo Cl\'emente, George Z. Xue, Amin Abyaneh, Liheng Ma, Rui Heng Yang, Tongtong Cao, Zhanguang Zhang, Yingxue Zhang

arXiv:2611.08156PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

La-JEPA is a compact world-action model whose policy transformer is trained from scratch in the feature space of a pretrained V-JEPA 2.1 encoder. It uses WAM-style joint training to predict future features with actions, but generates only actions at inference. Among policies without embodied pretraining, the authors report state-of-the-art performance on clean/randomized RoboTwin 2.0 and LIBERO-Plus (with corrected instructions), with far fewer parameters and faster policy-call latency than Fast-WAM at comparable performance.

Why it matters

The authors address whether generation-oriented backbones are necessary for competitive robot control. They show that a compact action transformer trained from scratch (without a generative pretrained policy backbone) can outperform closest JEPA-based methods and match embodied-pretrained baselines on real-robot tasks, while also being faster than Fast-WAM. They further quantify how JEPA features and training-time future-prediction choices affect policy performance, and correct a LIBERO-Plus instruction corruption issue.

Method

  • Train a compact policy transformer from scratch in the feature space of a pretrained V-JEPA 2.1 encoder; World DiT predicts future features and Action DiT generates action chunks jointly during training, generating only actions at inference.
  • Use WAM-style attention masking so action tokens attend only to observations and actions (not future tokens), while predictive supervision shapes shared World DiT parameters during training.
  • Adopt diffusion transformers trained with future and action-space flow matching; at inference, drop future tokens, cache observation keys/values, and run only the Action DiT denoising while executing a fixed number of actions per replan.

Limitation

The authors report that the officially released LIBERO-Plus evaluator corrupts most instructions (appends perturbation parameters), which can obscure robustness and requires re-evaluating baselines under a single corrected “clean” protocol.

Abstract (from arXiv)

World-action models adapt pretrained video generators, and vision-language-action models build on pretrained vision-language backbones; both tie policy architecture and inference cost to large transformers designed for generation rather than control. It remains unclear how much these generation-oriented choices affect control. In this work, we introduce Latent Action JEPA (La-JEPA), a compact world-action model whose policy transformer is trained from scratch in the feature space of a pretrained V-JEPA 2.1 encoder. Following Fast-WAM, La-JEPA learns to predict future features jointly with actions during training and generates only actions at inference. Among policies without embodied pretraining, La-JEPA achieves state-of-the-art performance in clean and randomized RoboTwin 2.0 scenes and on LIBERO-Plus with correct instructions, while using far fewer parameters than the strongest competing policies. It runs substantially faster than Fast-WAM at comparable performance and outperforms the closest JEPA-based methods, LeapBot-WA and JEPA-WAM, on LIBERO-Plus and RoboTwin 2.0 without their pretrained policy backbones. On four real-robot tasks, its average success is comparable to pi0.5 and above non-embodied-pretrained baselines. Through a design study of the choices behind La-JEPA, we show that JEPA features outperform video VAE latents, most clearly out of distribution, and that training-time future prediction choices significantly impact the policy even when future prediction is skipped at inference. We correct the LIBERO-Plus instructions, as the officially released evaluator corrupts them for most variants, and re-evaluate several strong open baselines. Without a generative pretrained policy backbone, La-JEPA is better and faster than far larger policies, suggesting that robot foundation models may benefit from architectures designed for control rather than inherited from generation.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-05 · Indexed by Pub-AI: 2026-11-06 UTC

preprint

La-JEPA: Compact Robot Policies with Latent World Modeling

Mateo Clémente, George Z. Xue, Amin Abyaneh, Liheng Ma, Rui Heng Yang, Tongtong Cao, Zhanguang Zhang, Yingxue Zhang

Preprint posted: 2026-11-05 · Original record ↗

arxiv
2611.08156
Explicit source versions (1)
  • v1 · 2026-11-05
  1. Preprint available

    Source date: 2026-11-05 · First detected: 2026-11-06 UTC · Historical baseline

    Source evidence (3)
Abstract · arxiv

World-action models adapt pretrained video generators, and vision-language-action models build on pretrained vision-language backbones; both tie policy architecture and inference cost to large transformers designed for generation rather than control. It remains unclear how much these generation-oriented choices affect control. In this work, we introduce Latent Action JEPA (La-JEPA), a compact world-action model whose policy transformer is trained from scratch in the feature space of a pretrained V-JEPA 2.1 encoder. Following Fast-WAM, La-JEPA learns to predict future features jointly with actions during training and generates only actions at inference. Among policies without embodied pretraining, La-JEPA achieves state-of-the-art performance in clean and randomized RoboTwin 2.0 scenes and on LIBERO-Plus with correct instructions, while using far fewer parameters than the strongest competing policies. It runs substantially faster than Fast-WAM at comparable performance and outperforms the closest JEPA-based methods, LeapBot-WA and JEPA-WAM, on LIBERO-Plus and RoboTwin 2.0 without their pretrained policy backbones. On four real-robot tasks, its average success is comparable to pi0.5 and above non-embodied-pretrained baselines. Through a design study of the choices behind La-JEPA, we show that JEPA features outperform video VAE latents, most clearly out of distribution, and that training-time future prediction choices significantly impact the policy even when future prediction is skipped at inference. We correct the LIBERO-Plus instructions, as the officially released evaluator corrupts them for most variants, and re-evaluate several strong open baselines. Without a generative pretrained policy backbone, La-JEPA is better and faster than far larger policies, suggesting that robot foundation models may benefit from architectures designed for control rather than inherited from generation.