4-bit NF4 quantized transformer inference with model CPU offload
Load the SD3.5 Large transformer in 4-bit NF4 quantized form (via BitsAndBytesConfig), plug it into the StableDiffusion3Pipeline, and use model CPU offload to reduce GPU memory usage during inference.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
SD3.5 Large is described as significantly larger than SD3 Medium, and the post emphasizes that memory optimization becomes important to make inference feasible on consumer hardware.
What changed
Instead of running the transformer at the default precision, the approach replaces the transformer weights with a bitsandbytes 4-bit NF4 quantized version and enables CPU offload on the pipeline.
Where it helps
- Running SD3.5 Large inference under tighter GPU memory constraints
- Using Diffusers with bitsandbytes NF4 quantization for transformer components
- Reducing memory use via CPU offload during generation
- Quantized model loading for lower-RAM machines
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The technique depends on having the required quantization libraries available and on choosing a compatible transformer submodule to quantize (as shown for SD3Transformer2DModel).
Sources
- Diffusers welcomes Stable Diffusion 3.5 Large ↗
Hugging Face · primary · Published 2024-10-22 · Checked 2026-11-10
