JEPA · 2026-10-14
Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
Yunxiang Luo, Constantino \'Alvarez Casado, Xiaoting Wu, Nhi Nguyen, Le Nguyen, Manuel Lage Ca\~nellas, Miguel Bordallo L\'opez
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors transfer frozen V-JEPA2 video features to a shared multimodal fusion model that handles missing sensors across RGB video, depth video, and radar time series. Using modality queries, fixed sensor slots, and explicit sensor availability, they classify breathing activity and pose for all seven non-empty sensor subsets on 39-subject OMuSense-23. With matched downstream parameter counts, all-view supervision yields activity Avg-7 macro-F1 of 75.23% ± 0.60.
Why it matters
The paper addresses how predictive video representations should be used when sensors are missing. It provides a controlled evaluation of subset-aware training, pooling capacity, and representation transfer (including radar) under declared sensor availability, rather than testing only complete-input performance. The results argue for direct supervision over known availability states with explicit availability checks.
Method
- Transfer frozen V-JEPA2 tokens (vjepa2-vitl-fpc64-256 encoder without predictor) to heterogeneous RGB, depth, and radar inputs, then cache adaptively averaged tokens for fusion.
- Use an availability-aware shared fusion transformer with modality queries, fixed sensor slots, and explicit presence/absence indicators; absent slots carry no reconstructed measurement.
- Train with controlled supervision regimes over non-empty sensor subsets (balanced vs all-view) and compare query pooling vs matched mean pooling; additionally test auxiliary complete-input objectives (Zero-loss/Consensus-JEPA).
Limitation
The authors state that they test declared absence, not failure detection or corrupted inputs, and that unseen sensors, incomplete records, free-living activity, new sites, and other backbones are not evaluated.
Abstract (from arXiv)
Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of $75.23%\pm0.60$ averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.
Related papers
- Phenomenon-Graph JEPA: Label-Efficient Representation Learning for Contactless Cardiorespiratory Sensing
- W-JEPA: Wavelet Joint-Embedding Predictive Architecture for Wearable Activity Recognition
- Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI
- MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures
- CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-11 · Indexed by Pub-AI: 2026-10-14 UTC
Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
Preprint posted: 2026-10-11 · Original record ↗
- arxiv
- 2610.15891
Explicit source versions (1)
- v1 · 2026-10-11
- Preprint available
Source date: 2026-10-11 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of $75.23%\pm0.60$ averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.
