Latent Dynamics · 2026-10-14
MDN-VJEPA: Multimodal Probabilistic Joint-Embedding World Models
Yongchao Huang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
MDN-VJEPA is a multimodal probabilistic world-model based on Joint-Embedding Predictive Architectures (JEPAs). It instantiates the VJEPA latent predictor as a Gaussian-mixture density network (MDN) to represent alternative futures with probabilities and within-mode uncertainty. It adds trajectory-level mode coherence and a distributional multi-horizon path-consistency regularizer for temporal coherence and Chapman–Kolmogorov consistency.
Why it matters
The authors argue that point predictions or unimodal Gaussian predictions can average together valid but distinct futures, producing invalid outcomes in planning. MDN-VJEPA aims to preserve the full multimodal predictive belief (distinct alternatives, their probabilities, and within-mode uncertainty), couple modes coherently across time, and encourage consistency between direct multi-horizon predictions and recursively composed rollouts.
Method
- Instantiates probabilistic JEPA (VJEPA) with a neural mixture density network: future latent distributions are Gaussian mixtures with component means, covariances, and mixture weights.
- Enforces trajectory-level mode coherence by selecting a categorical variable once over the prediction horizon so each mixture component parameterizes a whole future trajectory rather than independently per-horizon modes.
- Adds distributional path-consistency (Chapman–Kolmogorov-motivated) regularization by comparing direct versus recursively composed multi-horizon predictive distributions via a permutation-invariant sample discrepancy.
Abstract (from arXiv)
Joint-Embedding Predictive Architectures (JEPAs) learn world models by predicting future representations rather than reconstructing observations. Variational JEPA (VJEPA) gives this prediction probabilistic semantics and permits flexible conditional predictive families. We propose MDN-VJEPA, a multimodal probabilistic JEPA that instantiates the latent predictor as a mixture density network (MDN), representing future states with a Gaussian mixture whose components encode alternative futures, their probabilities, and within-mode uncertainty. For sequential prediction, we address two additional structural requirements: multi-modal futures should remain temporally coherent across a trajectory, and predictive beliefs should remain consistent when propagated to the same horizon through different temporal path compositions, as motivated by the Chapman--Kolmogorov equation. Together, these considerations define a three-level hierarchy of multimodal prediction, coherent multimodal trajectories, and compositionally consistent multimodal dynamics. Controlled experiments show that MDN-VJEPA closely recovers known branching predictive laws, that retaining the complete multimodal belief rather than its conditional mean can lead to materially different and substantially better planning decisions under branching futures, and that trajectory-level coupling and distributional consistency improve temporal coherence and multi-horizon agreement. These results support a simple principle: a probabilistic JEPA should preserve not only how uncertain the future is, but which distinct futures constitute that uncertainty, how they form coherent trajectories, and how those beliefs compose through time.
Related papers
- AVL-JEPA: Preventing Causal Dynamics Information Collapse In Joint Embedding Predictive Architecture World Models
- BeliefGraph-JEPA: Structured Latent World Models for Action-Conditioned Time Series
- Pythia: Toward Foundation World Models for Multimodal Time Series
- Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models
- EpicWorldModel: Exploration-driven Planning with Latent World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-08-21 · Indexed by Pub-AI: 2026-10-14 UTC
MDN-VJEPA: Multimodal Probabilistic Joint-Embedding World Models
Preprint posted: 2026-08-21 · Original record ↗
- arxiv
- 2610.13956
Explicit source versions (1)
- v1 · 2026-08-21
- Preprint available
Source date: 2026-08-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Joint-Embedding Predictive Architectures (JEPAs) learn world models by predicting future representations rather than reconstructing observations. Variational JEPA (VJEPA) gives this prediction probabilistic semantics and permits flexible conditional predictive families. We propose MDN-VJEPA, a multimodal probabilistic JEPA that instantiates the latent predictor as a mixture density network (MDN), representing future states with a Gaussian mixture whose components encode alternative futures, their probabilities, and within-mode uncertainty. For sequential prediction, we address two additional structural requirements: multi-modal futures should remain temporally coherent across a trajectory, and predictive beliefs should remain consistent when propagated to the same horizon through different temporal path compositions, as motivated by the Chapman--Kolmogorov equation. Together, these considerations define a three-level hierarchy of multimodal prediction, coherent multimodal trajectories, and compositionally consistent multimodal dynamics. Controlled experiments show that MDN-VJEPA closely recovers known branching predictive laws, that retaining the complete multimodal belief rather than its conditional mean can lead to materially different and substantially better planning decisions under branching futures, and that trajectory-level coupling and distributional consistency improve temporal coherence and multi-horizon agreement. These results support a simple principle: a probabilistic JEPA should preserve not only how uncertain the future is, but which distinct futures constitute that uncertainty, how they form coherent trajectories, and how those beliefs compose through time.