JEPA · 2026-11-11
RS-JEPA: Remote Sensing Joint-Embedding Predictive Architecture with Scan-Path Views for Cross-Sensor Image Retrieval
Tanishq Rachamalla, Srishti Kaushik, Muhammad Attique Khan, Gustau Camps-Valls, Swalpa Kumar Roy
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
RS-JEPA is a compact, label-free cross-sensor image retrieval method that uses a shared Vision Mamba encoder and scan-path views: two reading orders of the same image are treated as self-supervised views, predicting hidden patch features from visible ones. It adds Adversarial View Coupling (AVC) to make prediction harder and a neighbourhood consensus loss to align cross-sensor similarity geometry. The authors report the highest F1@5 among twelve re-implemented methods.
Why it matters
Cross-sensor remote sensing retrieval needs representations that capture shared scene content across heterogeneous sensors (e.g., SAR vs optical; PAN vs multispectral) without manual labels. RS-JEPA’s key idea is to derive self-supervised views from reading order in a state-space encoder, and to train retrieval geometry across sensors using neighbourhood consensus.
Method
- Shared Vision Mamba encoder for both sensors; per-sensor stems and predictors; train by predicting stop-gradient hidden patch features using scan-path views (different 2D patch traversals).
- Adversarial View Coupling (AVC): learn a hard split of visible patches into two views using a gradient-reversal scorer plus Coupled View Attention to refine cross-view context for prediction.
- Neighbourhood consensus loss: align same- and cross-sensor neighbourhood distributions (InfoNCE with in-batch negatives plus consensus over similarity geometry).
Limitation
RS-JEPA still needs co-registered training pairs, limiting scale. The authors test three benchmarks at 120×120 pixels, with no unseen regions or other sensors, one fixed scan-path set and a shared baseline budget; ablations are scored on test splits.
Abstract (from arXiv)
Cross-sensor remote sensing image retrieval aims to find scenes with the same content across heterogeneous sensors, such as SAR and optical or panchromatic (PAN) and multispectral (MS) imagery, without manual labels. Existing self-supervised methods either reconstruct sensor-specific pixel detail, align each image only with its co-registered twin, or rely on large transformers with 118-173M parameters (and up to 398M). We introduce RS-JEPA, a compact joint-embedding predictive framework built on a Vision Mamba encoder shared by both sensors. Since a state-space encoder reads an image as an ordered sequence of patches, we propose scan-path views, which treat two reading orders of the same image as two views with identical pixels and predict the hidden patches in one order from the visible patches in the other.To keep this prediction challenging, we propose Adversarial View Coupling (AVC), which learns the hardest split of the visible patches, and a neighbourhood consensus loss that aligns the scene similarities of both sensors. Extensive experiments on SAR-optical (BEN-14K, CBRSIR-VS) and PAN-MS (DSRSID) benchmarks demonstrate that RS-JEPA achieves the highest F1@5 among twelve re-implemented methods (72.92, 80.35 and 94.43) with a retrieval network of only 3.88M parameters (14.57M in training, 8-16times fewer than prior JEPA retrievers). Like prior methods, RS-JEPA trains on co-registered pairs. Our results highlight reading order as a source of self-supervised views. code: https://github.com/Tanishq251/RSJEPA.
Related papers
- CAPE-JEPA - Cross-modal Adversarial Probabilistic Embedding JEPA
- T-JEPA: A Temporal Joint-Embedding Predictive Architecture for Learning Better Remote Sensing Representations
- ChangeJEPA: Predict Before You Compare for Remote Sensing Change Detection
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- Enhancing Foundation Models for Imbalanced SAR Ship Classification via Targeted Oversampling
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-10 · Indexed by Pub-AI: 2026-11-11 UTC
RS-JEPA: Remote Sensing Joint-Embedding Predictive Architecture with Scan-Path Views for Cross-Sensor Image Retrieval
Preprint posted: 2026-11-10 · Original record ↗
- arxiv
- 2611.14496
Explicit source versions (1)
- v1 · 2026-11-10
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-11 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.14496v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Cross-sensor remote sensing image retrieval aims to find scenes with the same content across heterogeneous sensors, such as SAR and optical or panchromatic (PAN) and multispectral (MS) imagery, without manual labels. Existing self-supervised methods either reconstruct sensor-specific pixel detail, align each image only with its co-registered twin, or rely on large transformers with 118-173M parameters (and up to 398M). We introduce RS-JEPA, a compact joint-embedding predictive framework built on a Vision Mamba encoder shared by both sensors. Since a state-space encoder reads an image as an ordered sequence of patches, we propose scan-path views, which treat two reading orders of the same image as two views with identical pixels and predict the hidden patches in one order from the visible patches in the other.To keep this prediction challenging, we propose Adversarial View Coupling (AVC), which learns the hardest split of the visible patches, and a neighbourhood consensus loss that aligns the scene similarities of both sensors. Extensive experiments on SAR-optical (BEN-14K, CBRSIR-VS) and PAN-MS (DSRSID) benchmarks demonstrate that RS-JEPA achieves the highest F1@5 among twelve re-implemented methods (72.92, 80.35 and 94.43) with a retrieval network of only 3.88M parameters (14.57M in training, 8-16times fewer than prior JEPA retrievers). Like prior methods, RS-JEPA trains on co-registered pairs. Our results highlight reading order as a source of self-supervised views. code: https://github.com/Tanishq251/RSJEPA.
