JEPA · 2026-10-14

Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition

Yunxiang Luo, Constantino \'Alvarez Casado, Xiaoting Wu, Nhi Nguyen, Le Nguyen, Manuel Lage Ca\~nellas, Miguel Bordallo L\'opez

arXiv:2610.15891PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors transfer frozen V-JEPA2 video features to a shared multimodal fusion model that handles missing sensors across RGB video, depth video, and radar time series. Using modality queries, fixed sensor slots, and explicit sensor availability, they classify breathing activity and pose for all seven non-empty sensor subsets on 39-subject OMuSense-23. With matched downstream parameter counts, all-view supervision yields activity Avg-7 macro-F1 of 75.23% ± 0.60.

Why it matters

The paper addresses how predictive video representations should be used when sensors are missing. It provides a controlled evaluation of subset-aware training, pooling capacity, and representation transfer (including radar) under declared sensor availability, rather than testing only complete-input performance. The results argue for direct supervision over known availability states with explicit availability checks.

Method

  • Transfer frozen V-JEPA2 tokens (vjepa2-vitl-fpc64-256 encoder without predictor) to heterogeneous RGB, depth, and radar inputs, then cache adaptively averaged tokens for fusion.
  • Use an availability-aware shared fusion transformer with modality queries, fixed sensor slots, and explicit presence/absence indicators; absent slots carry no reconstructed measurement.
  • Train with controlled supervision regimes over non-empty sensor subsets (balanced vs all-view) and compare query pooling vs matched mean pooling; additionally test auxiliary complete-input objectives (Zero-loss/Consensus-JEPA).

Limitation

The authors state that they test declared absence, not failure detection or corrupted inputs, and that unseen sensors, incomplete records, free-living activity, new sites, and other backbones are not evaluated.

Abstract (from arXiv)

Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of $75.23%\pm0.60$ averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-11 · Indexed by Pub-AI: 2026-10-14 UTC

preprint

Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition

Yunxiang Luo, Constantino Álvarez Casado, Xiaoting Wu, Nhi Nguyen, Le Nguyen, Manuel Lage Cañellas, Miguel Bordallo López

Preprint posted: 2026-10-11 · Original record ↗

arxiv
2610.15891
Explicit source versions (1)
  • v1 · 2026-10-11
  1. Preprint available

    Source date: 2026-10-11 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of $75.23%\pm0.60$ averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.