JEPA · 2026-10-02
CF-JEPA: Improving Robustness of JEPA World Models via Controllability Factorization
Morgan Byrd, Robert Wright, Sehoon Ha
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
CF-JEPA is a JEPA-style latent world model that factorizes its latent space into an action-conditioned controllable part and an action-free uncontrollable part, aiming to capture distractor information in the uncontrollable region. The authors report comparable performance on nominal 2D/3D control tasks and improved robustness under visual distractors, where CF-JEPA is the only model that does not experience latent collapse. They also test a simulated robot task under distractors.
Why it matters
Vision-based control needs representations that ignore task-irrelevant background while staying controllable. The paper argues that standard JEPA world models (e.g., LeWM) are sensitive to distractors and can collapse, and shows that explicit controllability factorization can prevent collapse and improve control under distracted conditions, including when the action space requires perceiving the agent.
Method
- Learn an encoder and latent dynamics model from offline reward-free image transitions, with latent split z=[z^c,z^u]. Predict next latents with separate predictors: z^c uses action-conditioning; z^u is action-free.
- Use an inverse dynamics loss to shape controllability in z^c (predict action from [z_t^c,z_{t+1}^c]), plus adversarial dynamics with gradient reversal to penalize action predictability from z^u.
- Prevent latent collapse using SIGReg (as in LeWM) and plan in latent space using model-predictive control (CEM) with the controllable latent distance to the goal.
Limitation
Settings where the two interact, such as a person working next to the robot, would require extending the model because the method assumes that the uncontrollable part never affects the controllable part.
Abstract (from arXiv)
Controlling an agent with vision requires being able to separate useful information from irrelevant background information. JEPA-style latent world models seem like a natural approach for this, as they do not perform pixel-level reconstruction; however, they are still sensitive to these distractor signals and experience latent collapse. In this work, we introduce Controllability Factorized JEPA (CF-JEPA), a JEPA-style world model which splits the latent space into controllable and uncontrollable subspaces. This factorization allows us to capture all the distractor information into the uncontrollable region, while we use the control-relevant latent information for our task. With this, we show comparable performance across 2D and 3D control tasks under nominal conditions and improved performance under distracted conditions, where CF-JEPA is the only model that does not experience latent collapse. We also validate our model under distracted conditions for a simulated robot task, highlighting the practical application of such a scheme.
Related papers
- LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
- Hamiltonian JEPA: Action-Conditioned World Models with an Inherited Control State
- SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction
- Do-JEPA: From Masking to Intervention in Latent World Models
- Bilinear World Models: Learning Representations with Structured Dynamics for Efficient Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC
CF-JEPA: Improving Robustness of JEPA World Models via Controllability Factorization
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.00727
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Controlling an agent with vision requires being able to separate useful information from irrelevant background information. JEPA-style latent world models seem like a natural approach for this, as they do not perform pixel-level reconstruction; however, they are still sensitive to these distractor signals and experience latent collapse. In this work, we introduce Controllability Factorized JEPA (CF-JEPA), a JEPA-style world model which splits the latent space into controllable and uncontrollable subspaces. This factorization allows us to capture all the distractor information into the uncontrollable region, while we use the control-relevant latent information for our task. With this, we show comparable performance across 2D and 3D control tasks under nominal conditions and improved performance under distracted conditions, where CF-JEPA is the only model that does not experience latent collapse. We also validate our model under distracted conditions for a simulated robot task, highlighting the practical application of such a scheme.
