JEPA · 2026-10-21
Predictive Closure: A Set-Level Characterization of JEPA's Feature Selection
Mosh Levy, Aly Lidayan, Yoav Goldberg
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose “predictive closure,” a set-level characterization of how JEPA selects which features to encode. Instead of ranking features by individual predictability, predictive closure favors feasible feature sets that maximize predictive information (mutual information between current and next values). In controlled CIFAR-10 collage experiments with small latents, the authors report that unpredictable features can still be represented if they predict other represented features, and that steering can be indirect via predictive partners.
Why it matters
JEPA’s latent serves as the world model; features absent from it cannot inform planning or downstream tasks. The paper reframes feature selection from per-feature predictability to joint set properties, explaining when previously “unpredictable” features (e.g., a traffic light) can still be retained. It also provides concrete guidance for steering feature selection using auxiliary objectives or side information so that targeting one feature can strengthen others that form predictive relationships.
Method
- Reformulate JEPA’s objective to show that, among feasible feature sets constrained by latent regularization, the optimal represented set maximizes predictive information I_pred between current and next values.
- Decompose predictive information to characterize effects via aleatoric uncertainty and predictive leakage, arguing that features can be retained if they reduce predictive leakage by helping predict other selected features.
- Test selection predictions on synthetic CIFAR-10 collage sequences using frozen encoders and probes to measure whether target features are decodable under constrained latent capacity; also evaluate auxiliary reconstruction and predictor side-information steering.
Limitation
The authors note that testing requires sequences whose predictive relationships are known and can be changed one at a time, which natural data does not allow; therefore they construct synthetic sequences with controlled predictive relationships.
Abstract (from arXiv)
In joint-embedding predictive architectures (JEPA), an encoder and a predictor are trained together to produce a representation that minimizes the error in predicting the representation of the next observation. Because the representations of both the current and the next observation are learned, it is unclear which features they will include. Earlier characterizations describe the features JEPA selects in terms of their individual predictability, raising the concern that unpredictable features, like a traffic light, will not be encoded into the representation. However, the prediction error is also affected by the relationships between features, e.g., the light's color helps predict whether the traffic will stop. We propose the predictive closure characterization, which accounts for these relationships. Based on a reformulation of the JEPA training objective, we show that selection depends on properties of feature sets. This view agrees with earlier works when features evolve independently, but makes new predictions when they do not, for instance that a feature like the traffic light can still be retained. We test this characterization in controlled experiments, which show that unpredictable features can become better represented when they help predict other features, and that changing one feature's predictive relationships can affect how well others are represented. We also demonstrate implications for steering feature selection: targeting one feature with an auxiliary objective, or supplying it to the predictor as side information, can strengthen the representation of another feature whose future values it helps predict, so steering can be indirect.
Related papers
- Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures
- Does Joint-Embedding Predictive Architecture Pretraining Help Time Series Forecasting?
- CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
- Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models
- AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-21 UTC
Predictive Closure: A Set-Level Characterization of JEPA's Feature Selection
Preprint posted: 2026-10-19 · Original record ↗
- arxiv
- 2610.27364
Explicit source versions (1)
- v1 · 2026-10-19
- Preprint available
Source date: 2026-10-19 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · OAI arXiv created; not header datestamp (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
In joint-embedding predictive architectures (JEPA), an encoder and a predictor are trained together to produce a representation that minimizes the error in predicting the representation of the next observation. Because the representations of both the current and the next observation are learned, it is unclear which features they will include. Earlier characterizations describe the features JEPA selects in terms of their individual predictability, raising the concern that unpredictable features, like a traffic light, will not be encoded into the representation. However, the prediction error is also affected by the relationships between features, e.g., the light's color helps predict whether the traffic will stop. We propose the predictive closure characterization, which accounts for these relationships. Based on a reformulation of the JEPA training objective, we show that selection depends on properties of feature sets. This view agrees with earlier works when features evolve independently, but makes new predictions when they do not, for instance that a feature like the traffic light can still be retained. We test this characterization in controlled experiments, which show that unpredictable features can become better represented when they help predict other features, and that changing one feature's predictive relationships can affect how well others are represented. We also demonstrate implications for steering feature selection: targeting one feature with an auxiliary objective, or supplying it to the predictor as side information, can strengthen the representation of another feature whose future values it helps predict, so steering can be indirect.