JEPA · 2026-11-05

CAPE-JEPA - Cross-modal Adversarial Probabilistic Embedding JEPA

Aniket Junghare, Biplab Banerjee

arXiv:2611.06247PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

CAPE-JEPA is a probabilistic cross-modal JEPA for remote sensing cross-modal image retrieval. It replaces deterministic latent prediction with a cross-attentional Gaussian predictor (per-patch mean/variance) trained by negative log-likelihood, adds adversarial cross-modal distributional alignment via worst-case Epps–Pulley tests without explicit sample pairing, and uses in-training InfoNCE with attention-pooled heads for scene-level discrimination.

Why it matters

The authors target two limitations in cross-modal JEPA: deterministic prediction under inherent cross-sensor uncertainty and global modality-distribution mismatch that degrades retrieval geometry. Their approach combines uncertainty-aware probabilistic latent prediction, adversarial distributional regularization, and discriminative grounding, evaluated on BEN-14K and fMoW across SAR/multispectral/RGB modalities.

Method

  • Cross-modal probabilistic prediction: a cross-attentional network outputs per-patch Gaussian parameters (μ, σ^2) conditioned on opposite-modality context, trained with negative log-likelihood to model prediction uncertainty.
  • Adversarial distributional regularization: unimodal isotropic Gaussian regularization plus cross-modal adversarial variant aligning modality distributions using worst-case Epps–Pulley characteristic function tests, without explicit sample pairing.
  • In-training contrastive grounding: an InfoNCE objective with attention-pooled heads for scene-level discriminative grounding.
Abstract (from arXiv)

Cross-modal remote sensing image retrieval (RS-CMIR) requires representations that are simultaneously modality-invariant and semantically discriminative, despite the profound physical gap between heterogeneous sensors such as Synthetic Aperture Radar (SAR) and multispectral imagery. Recent JEPA-based Predictive self-supervised approaches such as X-JEPA formulate cross-modal alignment as a deterministic latent forecasting task, implicitly assuming uniform patch predictability and leaving cross-modal distributional alignment unaddressed. We propose CAPE-JEPA, a probabilistic cross-modal Joint Embedding Predictive Architecture with three contributions: (i) a cross-attentional Gaussian predictor outputting per-patch parameters (mu, sigma^2) conditioned on opposite-modality context, trained via negative log-likelihood to model inherent cross-sensor prediction uncertainty; (ii) adversarial distributional regularization combining unimodal isotropic Gaussian regularization with a novel cross-modal adversarial variant that aligns modality distributions via worst-case Epps-Pulley characteristic function tests without explicit sample pairing; and (iii) in-training contrastive learning with attention-pooled heads for scene-level discriminative grounding. Evaluated on BigEarthNet-14K (BEN-14K) and fMoW across SAR, multispectral, and RGB modalities, CAPE-JEPA surpasses X-JEPA by up to 13.74 points on BEN-14K and 15.59 points on fMoW, outperforming all baselines including SkySense and CROMA across both unimodal and cross-modal directions while using fewer parameters (407.30M vs. 432.72M).

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-05 · Indexed by Pub-AI: 2026-11-05 UTC

preprint

CAPE-JEPA - Cross-modal Adversarial Probabilistic Embedding JEPA

Aniket Junghare, Biplab Banerjee

First announced by arXiv: 2026-11-05 · Original record ↗

arxiv
2611.06247
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-05 · First detected: 2026-11-05 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.06247v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Cross-modal remote sensing image retrieval (RS-CMIR) requires representations that are simultaneously modality-invariant and semantically discriminative, despite the profound physical gap between heterogeneous sensors such as Synthetic Aperture Radar (SAR) and multispectral imagery. Recent JEPA-based Predictive self-supervised approaches such as X-JEPA formulate cross-modal alignment as a deterministic latent forecasting task, implicitly assuming uniform patch predictability and leaving cross-modal distributional alignment unaddressed. We propose CAPE-JEPA, a probabilistic cross-modal Joint Embedding Predictive Architecture with three contributions: (i) a cross-attentional Gaussian predictor outputting per-patch parameters (mu, sigma^2) conditioned on opposite-modality context, trained via negative log-likelihood to model inherent cross-sensor prediction uncertainty; (ii) adversarial distributional regularization combining unimodal isotropic Gaussian regularization with a novel cross-modal adversarial variant that aligns modality distributions via worst-case Epps-Pulley characteristic function tests without explicit sample pairing; and (iii) in-training contrastive learning with attention-pooled heads for scene-level discriminative grounding. Evaluated on BigEarthNet-14K (BEN-14K) and fMoW across SAR, multispectral, and RGB modalities, CAPE-JEPA surpasses X-JEPA by up to 13.74 points on BEN-14K and 15.59 points on fMoW, outperforming all baselines including SkySense and CROMA across both unimodal and cross-modal directions while using fewer parameters (407.30M vs. 432.72M).