Latent Dynamics · 2026-11-09

A Stochastic Latent Variable Generalization of Joint-Embedding Predictive Architectures

Nikos Giakoumoglou, Paschalis Giakoumoglou, Tania Stathaki

arXiv:2611.09302PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors introduce S-JEPA, a stochastic latent-variable generalization of Joint-Embedding Predictive Architectures (JEPAs). Unlike prior work that uses a deterministic token for the auxiliary variable z, S-JEPA uses a learned prior and posterior (CVAE-style) over z in representation space for masked image prediction with no decoders and no reconstruction. Stochasticity yields predictive uncertainty and diverse completions at no cost to representation quality.

Why it matters

The paper targets ambiguity in masked image prediction: a deterministic auxiliary variable collapses multi-modal plausible completions. By modeling z as a stochastic latent variable with a context-conditioned prior for inference, S-JEPA aims to recover the intended JEPA blueprint and provide uncertainty and diversity where deterministic predictors cannot.

Method

  • Replace deterministic auxiliary variable z in JEPA with a stochastic latent variable learned via CVAE-style training in representation space, using a posterior q(z|sx,sy) and context-conditioned prior p(z|sx).
  • Train by maximizing a variational lower bound (ELBO) with a prediction term (latent-space MSE with stop-gradient on the EMA target encoder) plus a KL term D_KL(q(z|sx,sy) || p(z|sx)) as an information bottleneck preventing target-conditioned shortcuts.
  • At inference, sample z from the learned context-conditioned prior p(z|sx) to produce diverse completions and spatially structured predictive uncertainty, without decoders or reconstruction.

Limitation

The authors state that their formulation is probabilistic but non-generative by design: S-JEPA has no decoder/reconstruction term, and the learning signal arrives entirely from the frozen target representation sy.

Abstract (from arXiv)

Joint-Embedding Predictive Architectures (JEPAs) predict the latent embedding of a target signal from a context signal, with an auxiliary variable z conditioning the predictor. Despite its prominent role in the original JEPA blueprint, the content of z has been carried by a fixed deterministic token in every prior instantiation, collapsing the genuinely multi-modal distribution over plausible completions to a point estimate. We present S-JEPA, which realises z as a proper stochastic latent variable for masked image prediction: a learned posterior infers z from context and target during training, while a learned context-conditioned prior approximates the distribution of plausible completions at inference, with no decoders and no reconstruction. In its deterministic limit the objective reduces to the I-JEPA loss in functional form, generalising that objective rather than strictly containing the I-JEPA model. Our central finding is that stochasticity adds capabilities, error-correlated predictive uncertainty and diverse completions, at no cost to representation quality: under matched architectures and schedules S-JEPA improves over the deterministic I-JEPA baseline and exceeds LeJEPA at matched architecture, while additionally producing spatially structured uncertainty maps and semantically diverse completions, neither of which the deterministic predictor produces.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-09 · Indexed by Pub-AI: 2026-11-09 UTC

preprint

A Stochastic Latent Variable Generalization of Joint-Embedding Predictive Architectures

Nikos Giakoumoglou, Paschalis Giakoumoglou, Tania Stathaki

First announced by arXiv: 2026-11-09 · Original record ↗

arxiv
2611.09302
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-09 · First detected: 2026-11-09 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.09302v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Joint-Embedding Predictive Architectures (JEPAs) predict the latent embedding of a target signal from a context signal, with an auxiliary variable z conditioning the predictor. Despite its prominent role in the original JEPA blueprint, the content of z has been carried by a fixed deterministic token in every prior instantiation, collapsing the genuinely multi-modal distribution over plausible completions to a point estimate. We present S-JEPA, which realises z as a proper stochastic latent variable for masked image prediction: a learned posterior infers z from context and target during training, while a learned context-conditioned prior approximates the distribution of plausible completions at inference, with no decoders and no reconstruction. In its deterministic limit the objective reduces to the I-JEPA loss in functional form, generalising that objective rather than strictly containing the I-JEPA model. Our central finding is that stochasticity adds capabilities, error-correlated predictive uncertainty and diverse completions, at no cost to representation quality: under matched architectures and schedules S-JEPA improves over the deterministic I-JEPA baseline and exceeds LeJEPA at matched architecture, while additionally producing spatially structured uncertainty maps and semantically diverse completions, neither of which the deterministic predictor produces.