JEPA · 2026-11-05
CAPE-JEPA - Cross-modal Adversarial Probabilistic Embedding JEPA
Aniket Junghare, Biplab Banerjee
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
CAPE-JEPA is a probabilistic cross-modal JEPA for remote sensing cross-modal image retrieval. It replaces deterministic latent prediction with a cross-attentional Gaussian predictor (per-patch mean/variance) trained by negative log-likelihood, adds adversarial cross-modal distributional alignment via worst-case Epps–Pulley tests without explicit sample pairing, and uses in-training InfoNCE with attention-pooled heads for scene-level discrimination.
Why it matters
The authors target two limitations in cross-modal JEPA: deterministic prediction under inherent cross-sensor uncertainty and global modality-distribution mismatch that degrades retrieval geometry. Their approach combines uncertainty-aware probabilistic latent prediction, adversarial distributional regularization, and discriminative grounding, evaluated on BEN-14K and fMoW across SAR/multispectral/RGB modalities.
Method
- Cross-modal probabilistic prediction: a cross-attentional network outputs per-patch Gaussian parameters (μ, σ^2) conditioned on opposite-modality context, trained with negative log-likelihood to model prediction uncertainty.
- Adversarial distributional regularization: unimodal isotropic Gaussian regularization plus cross-modal adversarial variant aligning modality distributions using worst-case Epps–Pulley characteristic function tests, without explicit sample pairing.
- In-training contrastive grounding: an InfoNCE objective with attention-pooled heads for scene-level discriminative grounding.
Abstract (from arXiv)
Cross-modal remote sensing image retrieval (RS-CMIR) requires representations that are simultaneously modality-invariant and semantically discriminative, despite the profound physical gap between heterogeneous sensors such as Synthetic Aperture Radar (SAR) and multispectral imagery. Recent JEPA-based Predictive self-supervised approaches such as X-JEPA formulate cross-modal alignment as a deterministic latent forecasting task, implicitly assuming uniform patch predictability and leaving cross-modal distributional alignment unaddressed. We propose CAPE-JEPA, a probabilistic cross-modal Joint Embedding Predictive Architecture with three contributions: (i) a cross-attentional Gaussian predictor outputting per-patch parameters (mu, sigma^2) conditioned on opposite-modality context, trained via negative log-likelihood to model inherent cross-sensor prediction uncertainty; (ii) adversarial distributional regularization combining unimodal isotropic Gaussian regularization with a novel cross-modal adversarial variant that aligns modality distributions via worst-case Epps-Pulley characteristic function tests without explicit sample pairing; and (iii) in-training contrastive learning with attention-pooled heads for scene-level discriminative grounding. Evaluated on BigEarthNet-14K (BEN-14K) and fMoW across SAR, multispectral, and RGB modalities, CAPE-JEPA surpasses X-JEPA by up to 13.74 points on BEN-14K and 15.59 points on fMoW, outperforming all baselines including SkySense and CROMA across both unimodal and cross-modal directions while using fewer parameters (407.30M vs. 432.72M).
Related papers
- RS-JEPA: Remote Sensing Joint-Embedding Predictive Architecture with Scan-Path Views for Cross-Sensor Image Retrieval
- T-JEPA: A Temporal Joint-Embedding Predictive Architecture for Learning Better Remote Sensing Representations
- Enhancing Foundation Models for Imbalanced SAR Ship Classification via Targeted Oversampling
- ChangeJEPA: Predict Before You Compare for Remote Sensing Change Detection
- A Stochastic Latent Variable Generalization of Joint-Embedding Predictive Architectures
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-05 · Indexed by Pub-AI: 2026-11-05 UTC
CAPE-JEPA - Cross-modal Adversarial Probabilistic Embedding JEPA
First announced by arXiv: 2026-11-05 · Original record ↗
- arxiv
- 2611.06247
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-05 · First detected: 2026-11-05 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.06247v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Cross-modal remote sensing image retrieval (RS-CMIR) requires representations that are simultaneously modality-invariant and semantically discriminative, despite the profound physical gap between heterogeneous sensors such as Synthetic Aperture Radar (SAR) and multispectral imagery. Recent JEPA-based Predictive self-supervised approaches such as X-JEPA formulate cross-modal alignment as a deterministic latent forecasting task, implicitly assuming uniform patch predictability and leaving cross-modal distributional alignment unaddressed. We propose CAPE-JEPA, a probabilistic cross-modal Joint Embedding Predictive Architecture with three contributions: (i) a cross-attentional Gaussian predictor outputting per-patch parameters (mu, sigma^2) conditioned on opposite-modality context, trained via negative log-likelihood to model inherent cross-sensor prediction uncertainty; (ii) adversarial distributional regularization combining unimodal isotropic Gaussian regularization with a novel cross-modal adversarial variant that aligns modality distributions via worst-case Epps-Pulley characteristic function tests without explicit sample pairing; and (iii) in-training contrastive learning with attention-pooled heads for scene-level discriminative grounding. Evaluated on BigEarthNet-14K (BEN-14K) and fMoW across SAR, multispectral, and RGB modalities, CAPE-JEPA surpasses X-JEPA by up to 13.74 points on BEN-14K and 15.59 points on fMoW, outperforming all baselines including SkySense and CROMA across both unimodal and cross-modal directions while using fewer parameters (407.30M vs. 432.72M).
