JEPA · 2026-10-26
Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction
Nilotpal Kakati
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study masked latent prediction (I-JEPA) by training small ViT encoders in a simulator with known image factors. They probe how readable a factor shared by two views remains after training. With the default (smaller) predictor, shared-factor readability is about 40% of SimCLR’s. Widening the predictor to match the encoder reversed the ordering against SimCLR and BYOL. They release the simulator and trained models.
Why it matters
The paper argues that what masked latent prediction retains from shared factors depends strongly on predictor capacity. Using a simulator where ground-truth factor values are known, the authors directly measure factor “readability” from frozen encoders, showing that sharing across views does not guarantee preservability under I-JEPA with its default predictor. Predictor width (and depth) can change which objective keeps which factors readable, even when encoder architecture and training budget are matched.
Method
- Train ViT encoders on simulator-generated two-view pairs with the same underlying component positions/energies and varying whether a studied factor (shape) is shared/redrawn/constant.
- Use masked latent prediction (I-JEPA) and compare against SimCLR and BYOL, trained on the same views with the same encoder architecture and training budget; vary I-JEPA predictor width/depth while keeping other settings fixed.
- After training, freeze encoders and fit a probe to predict known component parameters from encoder tokens; score readability as probe improvement over a median-error baseline.
Limitation
The authors state they “have not established why the predictor or the training data changed readability” and that their experiments cover “one factor in one simulator, with small encoders.” They also say they “cannot tell whether training removed the shape information or only made it harder to read.”
Abstract (from arXiv)
Masked latent prediction methods such as I-JEPA aim to learn predictable features, but what determines which information they retain is less clear. We built a simulator in which every factor behind an image is known, so a probe can measure how readable each factor remains after training. We trained small encoders (0.7M parameters) with I-JEPA, SimCLR and BYOL on the same pairs of views and probed a factor that both views share. I-JEPA had a predictor with half the encoder's width and depth, the proportions it uses for ViT-B. Its median score was about 40% of SimCLR's. Widening the predictor to the encoder's width reversed this ordering at every training seed, against both other methods. We release the simulator and the trained models.
Related papers
- JEPA Learns What the Mask Leaves Unrecoverable
- CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
- A Stochastic Latent Variable Generalization of Joint-Embedding Predictive Architectures
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- LEF-JEPA: Denoising-Trained Latent Energy Fields for Spatial Representation Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-23 · Indexed by Pub-AI: 2026-10-26 UTC
Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction
Preprint posted: 2026-10-23 · Original record ↗
- arxiv
- 2610.32562
Explicit source versions (1)
- v1 · 2026-10-23
- Preprint available
Source date: 2026-10-23 · First detected: 2026-10-27 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · atom:published (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Masked latent prediction methods such as I-JEPA aim to learn predictable features, but what determines which information they retain is less clear. We built a simulator in which every factor behind an image is known, so a probe can measure how readable each factor remains after training. We trained small encoders (0.7M parameters) with I-JEPA, SimCLR and BYOL on the same pairs of views and probed a factor that both views share. I-JEPA had a predictor with half the encoder's width and depth, the proportions it uses for ViT-B. Its median score was about 40% of SimCLR's. Widening the predictor to the encoder's width reversed this ordering at every training seed, against both other methods. We release the simulator and the trained models.
