JEPA · 2026-10-06
CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
Mintu Dutta, Ritesh Vyas, Mohendra Roy *
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
CI-JEPA extends I-JEPA with an intervention-aware counterfactual branch that predicts the representation change ΔZ between an original image and a modified one. On Flowers102, using frozen-encoder linear probing, CI-JEPA reports a best validation accuracy of 78.14% vs 77.55% for pretrained ViT-B/16 and I-JEPA. It also reports larger mean L2 representation changes for flower-center occlusion than for background blur/tint.
Why it matters
The authors argue that I-JEPA’s masked-region prediction objective does not explicitly model responses to specified visual interventions. CI-JEPA provides a way to study how latent representations change under semantic vs nuisance edits, using controlled counterfactual images and metrics like mean L2 change and Selective Intervention Ratio.
Method
- Add a counterfactual branch to I-JEPA: predict ΔZ = ZCF − Z, where Z is from the original image and ZCF from a counterfactual modified image.
- Condition the counterfactual predictor on an intervention embedding Eint and a binary intervention mask, and train with a Smooth L1 loss for the target representation change.
- Evaluate robustness as selective sensitivity: measure representation distances (including mean L2 changes) under semantic (flower-center occlusion) vs nuisance (background blur, background tint) interventions.
Limitation
Intervention effects are not reported for the baseline encoders, so these results do not establish improved selective sensitivity over the baselines; additionally, the best validation accuracies are not held-out test results and the 0.59-percentage-point gain should not be interpreted as statistically significant without repeated-run uncertainty estimates.
Abstract (from arXiv)
Self-supervised visual representation learning learns useful features without manual annotations during representation training. The image-based joint-embedding predictive architecture (I-JEPA) predicts latent representations of masked image regions, but its objective does not explicitly model responses to specified visual interventions. We introduce CI-JEPA, a counterfactual intervention-aware extension that learns to predict the representation change $\Delta Z = Z_{\mathrm{CF}} - Z$ between an original image and a modified counterpart. We assess representation robustness through selective sensitivity: stronger responses to task-relevant semantic changes than to nuisance changes. Experiments on Flowers102 use flower-center occlusion as a candidate semantic intervention and background blur and tint as candidate nuisance interventions. With frozen-encoder linear probing, CI-JEPA achieves a best validation accuracy of 78.14\%, compared with 77.55\% for both the pretrained ViT-B/16 and the I-JEPA baseline, a gain of 0.59 percentage points. The reported mean $L_2$ representation changes are 4.42 for center occlusion, 3.48 for background tint, and 2.83 for background blur. This ordering is consistent with relative semantic selectivity for the evaluated interventions, rather than complete nuisance invariance. The accuracy comparison is complementary and does not establish improved robustness over the baselines. These controlled image modifications provide a framework for studying intervention-induced changes in JEPA representations; they do not establish causal feature discovery or robustness to all visual changes.
Related papers
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- JEPA Learns What the Mask Leaves Unrecoverable
- Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures
- Revisiting Feature Prediction for Learning Visual Representations from Video
- VR-JEPA: Learning Contrastive-State Latent Guidance for Generation-based Video Reasoning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-04 · Indexed by Pub-AI: 2026-10-06 UTC
CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
Preprint posted: 2026-10-04 · Original record ↗
- arxiv
- 2610.05043
Explicit source versions (1)
- v1 · 2026-10-04
- Preprint available
Source date: 2026-10-04 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Self-supervised visual representation learning learns useful features without manual annotations during representation training. The image-based joint-embedding predictive architecture (I-JEPA) predicts latent representations of masked image regions, but its objective does not explicitly model responses to specified visual interventions. We introduce CI-JEPA, a counterfactual intervention-aware extension that learns to predict the representation change $ΔZ = Z_{\mathrm{CF}} - Z$ between an original image and a modified counterpart. We assess representation robustness through selective sensitivity: stronger responses to task-relevant semantic changes than to nuisance changes. Experiments on Flowers102 use flower-center occlusion as a candidate semantic intervention and background blur and tint as candidate nuisance interventions. With frozen-encoder linear probing, CI-JEPA achieves a best validation accuracy of 78.14\%, compared with 77.55\% for both the pretrained ViT-B/16 and the I-JEPA baseline, a gain of 0.59 percentage points. The reported mean $L_2$ representation changes are 4.42 for center occlusion, 3.48 for background tint, and 2.83 for background blur. This ordering is consistent with relative semantic selectivity for the evaluated interventions, rather than complete nuisance invariance. The accuracy comparison is complementary and does not establish improved robustness over the baselines. These controlled image modifications provide a framework for studying intervention-induced changes in JEPA representations; they do not establish causal feature discovery or robustness to all visual changes.