On-the-fly de-palettization with 6-bit weight palettization in Core ML
Compress Stable Diffusion model weights using 6-bit weight palettization (a form of quantization) during conversion to the Core ML format, then keep weights compressed as palette indices and convert (de-palettize) them on the fly as inference progresses layer by layer rather than fully uncompressing at model load time.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
This reduces model storage and download/disk footprint and lowers memory use during inference. The source claims that using less memory on the critical execution path (memory transfers) can make on-device inference faster compared with a fully uncompressed-at-load approach.
What changed
Previously, Core ML uncompressed palletized weights when the model was first loaded, so runtime memory matched the uncompressed size. The described improvement keeps weights stored as 6-bit values and performs the conversion on demand while running inference across layers.
Where it helps
- Running Stable Diffusion variants on-device with reduced memory footprint
- Packaging Core ML models for native apps while minimizing app/model size
- Converting pre-trained Stable Diffusion models to Core ML with quantized weights
- Reducing download time by shrinking on-device model storage needs
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The compressed palette-index weights cannot be used directly for computation; they must be converted during inference. Also, the source indicates that using 6-bit models requires newer Core ML framework versions (iOS/iPadOS 17 or macOS 14).
Sources
- Faster Stable Diffusion with Core ML on iPhone, iPad, and Mac ↗
Hugging Face · primary · Published 2023-06-15 · Checked 2026-11-10
