← AI Techniques Radar
Inference efficiency

On-the-fly de-palettization with 6-bit weight palettization in Core ML

Compress Stable Diffusion model weights using 6-bit weight palettization (a form of quantization) during conversion to the Core ML format, then keep weights compressed as palette indices and convert (de-palettize) them on the fly as inference progresses layer by layer rather than fully uncompressing at model load time.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

This reduces model storage and download/disk footprint and lowers memory use during inference. The source claims that using less memory on the critical execution path (memory transfers) can make on-device inference faster compared with a fully uncompressed-at-load approach.

What changed

Previously, Core ML uncompressed palletized weights when the model was first loaded, so runtime memory matched the uncompressed size. The described improvement keeps weights stored as 6-bit values and performs the conversion on demand while running inference across layers.

Where it helps

  • Running Stable Diffusion variants on-device with reduced memory footprint
  • Packaging Core ML models for native apps while minimizing app/model size
  • Converting pre-trained Stable Diffusion models to Core ML with quantized weights
  • Reducing download time by shrinking on-device model storage needs

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

The compressed palette-index weights cannot be used directly for computation; they must be converted during inference. Also, the source indicates that using 6-bit models requires newer Core ML framework versions (iOS/iPadOS 17 or macOS 14).

Sources