Quantization-aware training plus auxiliary knowledge distillation stacked with token merging for efficient Stable Diffusion UNet inference on CPU
Train a Stable Diffusion pipeline with quantization-aware training (QAT) using NNCF, augment the training objective with knowledge distillation (teacher-student setup using the source model as teacher), stabilize training with EMA of parameters (excluding quantizers), and—during optimization—modify the attention UNet by applying token merging before self-attention and stack it with 8-bit quantization for further efficiency.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
The source claims that post-training quantization alone is insufficient for Stable Diffusion and proposes a training workflow that jointly tunes quantized weights while also using distillation and token merging to reduce inference latency and model footprint for CPU/edge resource constraints.
What changed
Instead of relying on straightforward post-training quantization, the workflow changes the training and architecture side by (1) integrating QAT for quantization parameters during training, (2) adding an auxiliary knowledge-distillation term with the original model as teacher, (3) using EMA for training stability while excluding quantizers, and (4) modifying attention computation via token merging that is made compatible with OpenVINO and applied on top of 8-bit quantization.
Where it helps
- Optimize Stable Diffusion for resource-constrained CPU/edge deployments
- Reducing latency and checkpoint size for text-to-image pipelines
- Training quantized UNet/attention components with distillation
- Implementing token-merging compatible attention in an optimized inference stack
- Adapting Stable Diffusion optimization workflow for OpenVINO-based runtime
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
Token merging reduces model capacity; the source notes a practical caveat that for more complex datasets, a lower merging ratio should be used. It also states that applying this approach generally requires fine-tuning/retraining to restore accuracy after quantization.
Sources
- Optimizing Stable Diffusion for Intel CPUs with NNCF and 🤗 Optimum ↗
Hugging Face · primary · Published 2023-05-25 · Checked 2026-11-10
