JEPA · 2026-10-30
Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI
Serwan Jassim, Victoria Bosch, Daniel Anthes, Elia Bruni, Tim Kietzmann
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors adapt JEPA to pre-train participant-specific fMRI encoders using brain-only data from NSD. Encoders never see stimulus images or stimulus-derived representations. They evaluate frozen encoders with MindEye2 for image reconstruction and CorText for brain captioning, including cross-participant generalization. The authors report that independently trained participant embeddings can be aligned fully unsupervised without shared stimuli, enabling cross-subject decoding without fine-tuning.
Why it matters
They target two bottlenecks in fMRI decoding: limited per-participant data and difficulty pooling across participants due to representational differences. The paper claims that latent prediction from repeated stimulus-evoked brain responses yields (i) expressive within-subject embeddings for generative decoding and (ii) an alignable geometry across subjects without shared stimuli or externally defined anchors, enabling transfer of a decoding head between participants without fine-tuning.
Method
- Pre-train one JEPA encoder per participant on NSD using only parcellated fMRI responses: predict latent representations of masked parcels from a different repetition of the same stimulus; no stimulus images or stimulus-derived targets are used during encoder training.
- Freeze the pre-trained brain encoder and train generative decoding heads for image reconstruction (MindEye2) and brain captioning (CorText) using the same targets as the downstream pipelines.
- Align independently trained participant encoders fully unsupervised: fit orthogonal maps per parcel token using only each participant’s own training embeddings, then transfer a captioning/reconstruction head to aligned representations without fine-tuning.
Limitation
The paper text provided does not report any explicit limitation.
Abstract (from arXiv)
Decoding sensory-evoked functional magnetic resonance imaging (fMRI) data is a central analytical tool in cognitive neuroscience. Here we ask whether self-supervised learning (SSL), which has been scarcely explored in this domain, can yield representations that support strong downstream performance. To do so, we adapt a joint embedding predictive architecture (JEPA) to train participant-specific encoders on brain data alone. The encoders never see the stimulus images or any representation derived from them. Applied to the Natural Scenes Dataset (NSD), we evaluate our learned brain embeddings on the downstream tasks of brain captioning and image reconstruction, as well as cross-participant generalization. Our self-supervised approach achieves competitive performance with parameter-rich supervised pipelines that are trained end-to-end with access to image and language data. We further find that encoders trained independently for different participants can be aligned fully unsupervised, using neither images nor stimuli shared between them. After alignment, a neural decoding head trained on one participant can be applied to another without fine-tuning and retains most of its within-subject performance on image reconstruction and brain captioning. By removing the need for images and shared stimuli, our approach is a step towards pooling fMRI data across participants and, eventually, across datasets.
Related papers
- CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
- NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex
- Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC
Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI
Preprint posted: 2026-10-29 · Original record ↗
- arxiv
- 2610.40280
Explicit source versions (1)
- v1 · 2026-10-29
- Preprint available
Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40280v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Decoding sensory-evoked functional magnetic resonance imaging (fMRI) data is a central analytical tool in cognitive neuroscience. Here we ask whether self-supervised learning (SSL), which has been scarcely explored in this domain, can yield representations that support strong downstream performance. To do so, we adapt a joint embedding predictive architecture (JEPA) to train participant-specific encoders on brain data alone. The encoders never see the stimulus images or any representation derived from them. Applied to the Natural Scenes Dataset (NSD), we evaluate our learned brain embeddings on the downstream tasks of brain captioning and image reconstruction, as well as cross-participant generalization. Our self-supervised approach achieves competitive performance with parameter-rich supervised pipelines that are trained end-to-end with access to image and language data. We further find that encoders trained independently for different participants can be aligned fully unsupervised, using neither images nor stimuli shared between them. After alignment, a neural decoding head trained on one participant can be applied to another without fine-tuning and retains most of its within-subject performance on image reconstruction and brain captioning. By removing the need for images and shared stimuli, our approach is a step towards pooling fMRI data across participants and, eventually, across datasets.