Vector-Quantized Discrete Latent Diffusion (VQ-Diffusion style)
Use a diffusion model whose forward noising and reverse denoising operate in a discrete latent token space. Images are first encoded into quantized token indices via a VQ-VAE/VQGAN-style encoder with a fixed codebook, and then a conditional encoder-decoder transformer learns to predict the categorical distribution of the un-noised latent tokens conditioned on a text prompt.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
Working in a discrete latent space reframes denoising as predicting categorical distributions over codebook entries rather than regressing continuous noise. This enables a transformer-based reverse process that can use global attention over the latent token grid while still being trained to output distributions for the fully un-noised latents.
What changed
Compared to continuous diffusion, the noising/denoising process is defined over quantized latent vectors (discrete codebook entries). The forward step allows latent tokens to stay the same, be resampled to another non-masked codebook vector, or become masked, and the reverse model predicts the distribution over un-noised latent tokens in one forward pass.
Where it helps
- Text-to-image generation using a conditional latent diffusion transformer
- Discrete sequence modeling over image latent tokens conditioned on prompts
- Comparative study between discrete diffusion and autoregressive transformer image models
- Implementations using diffusion pipelines that operate on tokenized latent spaces
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The approach depends on having a pre-trained quantized latent tokenizer (VQ-VAE/VQGAN); the quantizer is described as frozen during diffusion training. It also relies on the specific discrete forward-process design (stay/resample/mask transitions), which may require careful alignment to the target data and codebook.
Sources
- VQ-Diffusion ↗
Hugging Face · primary · Published 2022-11-30 · Checked 2026-11-10
