Video World Models · 2026-10-06
From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Model
Vicente Balmaseda, Ching-Long Lin, Tianbao Yang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
SCION trains a single pixel-space transformer to learn self-supervised representations (contrastive loss on clean augmentations) and to generate images via flow matching in one jointly optimized model. The shared encoder is conditioned on timestep and a conditioning embedding derived from either a learned null embedding (representation learning) or the image’s own [CLS] embedding (generation). A learned prior enables unconditional sampling without a reference image.
Why it matters
The authors address the mismatch between contrastive learning (favoring invariance on clean augmented views) and flow matching (requiring fine detail and spatial layout from noisy images). SCION’s single encoder supports both objectives through two conditioning modes and readouts ([CLS] for semantics, patch tokens for generation), with joint optimization strategies (gradient-norm balancing, stop-gradient, and a prior for reference-free sampling).
Method
- Uses a shared pixel-space JiT encoder conditioned on timestep and an embedding; readouts: [CLS] embedding for contrastive semantics and patch tokens for generation via a decoder.
- Representation training: encode strongly augmented views at t=1 with null conditioning; contrastive loss trains the [CLS] embedding; generation training: condition on the image’s own [CLS] embedding and predict clean images with a flow-matching objective.
- To sample without a reference image, jointly learn a prior over the embedding in the same run; employ gradient-norm balancing and stop-gradient mechanisms for joint optimization.
Abstract (from arXiv)
Strong image generation models are conditioned on class labels, aligned to frozen pretrained encoders, or built on separately trained autoencoders. While effective, generation then depends on supervision or pretraining: labels must be annotated, and encoders or autoencoders pretrained for the target domain. We study joint generative and self-supervised representation learning in a single model, enabling self-conditioned generation without labels or pretrained models. This is challenging because the objectives are mismatched: contrastive learning consumes clean augmented views and favors coarse, invariant semantics, while flow matching consumes noisy images and must preserve the fine detail and spatial layout that contrastive learning discards. We propose SCION (Self-conditioned Generation on Self-supervised representation), whose core is a single pixel-space encoder conditioned on the flow timestep and an embedding. For representation learning, this conditioning embedding is a learned global vector shared across images, with the encoder's [CLS] token yielding the semantic representation trained by the contrastive loss. For generative training, the conditioning embedding is the image's own [CLS] representation, while patch tokens pass through a decoder to predict the image. To sample without a reference image at inference, we jointly learn a prior over the embedding. Gradient-norm balancing and stop-gradient mechanisms enable joint optimization in one run. SCION is self-supervised and self-contained, with no labels or pretrained models. On ImageNet 256x256, with the JiT-B recipe and no representation guidance, SCION reaches 8.92 FID, surpassing class-unconditional iREPA, which aligns to pretrained DINOv2 (46.44), and RCG, which conditions on it (14.27). With JiT-L, SCION achieves 5.89 FID without guidance and 3.47 with representation guidance, outperforming RCG with the ADM recipe (6.24).
Related papers
- Are We Overthinking One-Step Generation?
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning
- CAST: Causal Advantage-Structured Training with Spatially Grounded Compositional Rewards for Diffusion Models
- CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-05 · Indexed by Pub-AI: 2026-10-06 UTC
From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Model
Preprint posted: 2026-10-05 · Original record ↗
- arxiv
- 2610.05711
Explicit source versions (1)
- v1 · 2026-10-05
- Preprint available
Source date: 2026-10-05 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Strong image generation models are conditioned on class labels, aligned to frozen pretrained encoders, or built on separately trained autoencoders. While effective, generation then depends on supervision or pretraining: labels must be annotated, and encoders or autoencoders pretrained for the target domain. We study joint generative and self-supervised representation learning in a single model, enabling self-conditioned generation without labels or pretrained models. This is challenging because the objectives are mismatched: contrastive learning consumes clean augmented views and favors coarse, invariant semantics, while flow matching consumes noisy images and must preserve the fine detail and spatial layout that contrastive learning discards. We propose SCION (Self-conditioned Generation on Self-supervised representation), whose core is a single pixel-space encoder conditioned on the flow timestep and an embedding. For representation learning, this conditioning embedding is a learned global vector shared across images, with the encoder's [CLS] token yielding the semantic representation trained by the contrastive loss. For generative training, the conditioning embedding is the image's own [CLS] representation, while patch tokens pass through a decoder to predict the image. To sample without a reference image at inference, we jointly learn a prior over the embedding. Gradient-norm balancing and stop-gradient mechanisms enable joint optimization in one run. SCION is self-supervised and self-contained, with no labels or pretrained models. On ImageNet 256x256, with the JiT-B recipe and no representation guidance, SCION reaches 8.92 FID, surpassing class-unconditional iREPA, which aligns to pretrained DINOv2 (46.44), and RCG, which conditions on it (14.27). With JiT-L, SCION achieves 5.89 FID without guidance and 3.47 with representation guidance, outperforming RCG with the ADM recipe (6.24).