← AI Techniques Radar
Training & fine-tuning

Quantization-aware training plus auxiliary knowledge distillation stacked with token merging for efficient Stable Diffusion UNet inference on CPU

Train a Stable Diffusion pipeline with quantization-aware training (QAT) using NNCF, augment the training objective with knowledge distillation (teacher-student setup using the source model as teacher), stabilize training with EMA of parameters (excluding quantizers), and—during optimization—modify the attention UNet by applying token merging before self-attention and stack it with 8-bit quantization for further efficiency.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

The source claims that post-training quantization alone is insufficient for Stable Diffusion and proposes a training workflow that jointly tunes quantized weights while also using distillation and token merging to reduce inference latency and model footprint for CPU/edge resource constraints.

What changed

Instead of relying on straightforward post-training quantization, the workflow changes the training and architecture side by (1) integrating QAT for quantization parameters during training, (2) adding an auxiliary knowledge-distillation term with the original model as teacher, (3) using EMA for training stability while excluding quantizers, and (4) modifying attention computation via token merging that is made compatible with OpenVINO and applied on top of 8-bit quantization.

Where it helps

  • Optimize Stable Diffusion for resource-constrained CPU/edge deployments
  • Reducing latency and checkpoint size for text-to-image pipelines
  • Training quantized UNet/attention components with distillation
  • Implementing token-merging compatible attention in an optimized inference stack
  • Adapting Stable Diffusion optimization workflow for OpenVINO-based runtime

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

Token merging reduces model capacity; the source notes a practical caveat that for more complex datasets, a lower merging ratio should be used. It also states that applying this approach generally requires fine-tuning/retraining to restore accuracy after quantization.

Sources