JEPA · 2026-11-11

RS-JEPA: Remote Sensing Joint-Embedding Predictive Architecture with Scan-Path Views for Cross-Sensor Image Retrieval

Tanishq Rachamalla, Srishti Kaushik, Muhammad Attique Khan, Gustau Camps-Valls, Swalpa Kumar Roy

arXiv:2611.14496PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

RS-JEPA is a compact, label-free cross-sensor image retrieval method that uses a shared Vision Mamba encoder and scan-path views: two reading orders of the same image are treated as self-supervised views, predicting hidden patch features from visible ones. It adds Adversarial View Coupling (AVC) to make prediction harder and a neighbourhood consensus loss to align cross-sensor similarity geometry. The authors report the highest F1@5 among twelve re-implemented methods.

Why it matters

Cross-sensor remote sensing retrieval needs representations that capture shared scene content across heterogeneous sensors (e.g., SAR vs optical; PAN vs multispectral) without manual labels. RS-JEPA’s key idea is to derive self-supervised views from reading order in a state-space encoder, and to train retrieval geometry across sensors using neighbourhood consensus.

Method

  • Shared Vision Mamba encoder for both sensors; per-sensor stems and predictors; train by predicting stop-gradient hidden patch features using scan-path views (different 2D patch traversals).
  • Adversarial View Coupling (AVC): learn a hard split of visible patches into two views using a gradient-reversal scorer plus Coupled View Attention to refine cross-view context for prediction.
  • Neighbourhood consensus loss: align same- and cross-sensor neighbourhood distributions (InfoNCE with in-batch negatives plus consensus over similarity geometry).

Limitation

RS-JEPA still needs co-registered training pairs, limiting scale. The authors test three benchmarks at 120×120 pixels, with no unseen regions or other sensors, one fixed scan-path set and a shared baseline budget; ablations are scored on test splits.

Abstract (from arXiv)

Cross-sensor remote sensing image retrieval aims to find scenes with the same content across heterogeneous sensors, such as SAR and optical or panchromatic (PAN) and multispectral (MS) imagery, without manual labels. Existing self-supervised methods either reconstruct sensor-specific pixel detail, align each image only with its co-registered twin, or rely on large transformers with 118-173M parameters (and up to 398M). We introduce RS-JEPA, a compact joint-embedding predictive framework built on a Vision Mamba encoder shared by both sensors. Since a state-space encoder reads an image as an ordered sequence of patches, we propose scan-path views, which treat two reading orders of the same image as two views with identical pixels and predict the hidden patches in one order from the visible patches in the other.To keep this prediction challenging, we propose Adversarial View Coupling (AVC), which learns the hardest split of the visible patches, and a neighbourhood consensus loss that aligns the scene similarities of both sensors. Extensive experiments on SAR-optical (BEN-14K, CBRSIR-VS) and PAN-MS (DSRSID) benchmarks demonstrate that RS-JEPA achieves the highest F1@5 among twelve re-implemented methods (72.92, 80.35 and 94.43) with a retrieval network of only 3.88M parameters (14.57M in training, 8-16times fewer than prior JEPA retrievers). Like prior methods, RS-JEPA trains on co-registered pairs. Our results highlight reading order as a source of self-supervised views. code: https://github.com/Tanishq251/RSJEPA.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-10 · Indexed by Pub-AI: 2026-11-11 UTC

preprint

RS-JEPA: Remote Sensing Joint-Embedding Predictive Architecture with Scan-Path Views for Cross-Sensor Image Retrieval

Tanishq Rachamalla, Srishti Kaushik, Muhammad Attique Khan, Gustau Camps-Valls, Swalpa Kumar Roy

Preprint posted: 2026-11-10 · Original record ↗

arxiv
2611.14496
Explicit source versions (1)
  • v1 · 2026-11-10
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-11 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Cross-sensor remote sensing image retrieval aims to find scenes with the same content across heterogeneous sensors, such as SAR and optical or panchromatic (PAN) and multispectral (MS) imagery, without manual labels. Existing self-supervised methods either reconstruct sensor-specific pixel detail, align each image only with its co-registered twin, or rely on large transformers with 118-173M parameters (and up to 398M). We introduce RS-JEPA, a compact joint-embedding predictive framework built on a Vision Mamba encoder shared by both sensors. Since a state-space encoder reads an image as an ordered sequence of patches, we propose scan-path views, which treat two reading orders of the same image as two views with identical pixels and predict the hidden patches in one order from the visible patches in the other.To keep this prediction challenging, we propose Adversarial View Coupling (AVC), which learns the hardest split of the visible patches, and a neighbourhood consensus loss that aligns the scene similarities of both sensors. Extensive experiments on SAR-optical (BEN-14K, CBRSIR-VS) and PAN-MS (DSRSID) benchmarks demonstrate that RS-JEPA achieves the highest F1@5 among twelve re-implemented methods (72.92, 80.35 and 94.43) with a retrieval network of only 3.88M parameters (14.57M in training, 8-16times fewer than prior JEPA retrievers). Like prior methods, RS-JEPA trains on co-registered pairs. Our results highlight reading order as a source of self-supervised views. code: https://github.com/Tanishq251/RSJEPA.