JEPA · 2026-10-26

Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction

Nilotpal Kakati

arXiv:2610.32562PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors study masked latent prediction (I-JEPA) by training small ViT encoders in a simulator with known image factors. They probe how readable a factor shared by two views remains after training. With the default (smaller) predictor, shared-factor readability is about 40% of SimCLR’s. Widening the predictor to match the encoder reversed the ordering against SimCLR and BYOL. They release the simulator and trained models.

Why it matters

The paper argues that what masked latent prediction retains from shared factors depends strongly on predictor capacity. Using a simulator where ground-truth factor values are known, the authors directly measure factor “readability” from frozen encoders, showing that sharing across views does not guarantee preservability under I-JEPA with its default predictor. Predictor width (and depth) can change which objective keeps which factors readable, even when encoder architecture and training budget are matched.

Method

  • Train ViT encoders on simulator-generated two-view pairs with the same underlying component positions/energies and varying whether a studied factor (shape) is shared/redrawn/constant.
  • Use masked latent prediction (I-JEPA) and compare against SimCLR and BYOL, trained on the same views with the same encoder architecture and training budget; vary I-JEPA predictor width/depth while keeping other settings fixed.
  • After training, freeze encoders and fit a probe to predict known component parameters from encoder tokens; score readability as probe improvement over a median-error baseline.

Limitation

The authors state they “have not established why the predictor or the training data changed readability” and that their experiments cover “one factor in one simulator, with small encoders.” They also say they “cannot tell whether training removed the shape information or only made it harder to read.”

Abstract (from arXiv)

Masked latent prediction methods such as I-JEPA aim to learn predictable features, but what determines which information they retain is less clear. We built a simulator in which every factor behind an image is known, so a probe can measure how readable each factor remains after training. We trained small encoders (0.7M parameters) with I-JEPA, SimCLR and BYOL on the same pairs of views and probed a factor that both views share. I-JEPA had a predictor with half the encoder's width and depth, the proportions it uses for ViT-B. Its median score was about 40% of SimCLR's. Widening the predictor to the encoder's width reversed this ordering at every training seed, against both other methods. We release the simulator and the trained models.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-23 · Indexed by Pub-AI: 2026-10-26 UTC

preprint

Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction

Nilotpal Kakati

Preprint posted: 2026-10-23 · Original record ↗

arxiv
2610.32562
Explicit source versions (1)
  • v1 · 2026-10-23
  1. Preprint available

    Source date: 2026-10-23 · First detected: 2026-10-27 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Masked latent prediction methods such as I-JEPA aim to learn predictable features, but what determines which information they retain is less clear. We built a simulator in which every factor behind an image is known, so a probe can measure how readable each factor remains after training. We trained small encoders (0.7M parameters) with I-JEPA, SimCLR and BYOL on the same pairs of views and probed a factor that both views share. I-JEPA had a predictor with half the encoder's width and depth, the proportions it uses for ViT-B. Its median score was about 40% of SimCLR's. Widening the predictor to the encoder's width reversed this ordering at every training seed, against both other methods. We release the simulator and the trained models.