Use JAX JIT with static shapes plus XLA parallelism (pmap) for SDXL TPU inference
Compile an SDXL image-generation pipeline in JAX with just-in-time (JIT) compilation that assumes fixed (static) input/output shapes, then scale generation across multiple TPU chips using pmap-style parallelism so multiple images render concurrently with the same compiled program.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
SDXL inference can be bottlenecked by compilation overhead and by how efficiently hardware can be used. This approach reduces repeated runtime work by compiling once for fixed shapes, and improves utilization by expressing single-program multiple-data parallel execution across TPU devices.
What changed
Instead of treating inference as fully dynamic per request, the pipeline is structured around constant tensor dimensions (including fixed prompt tensor shapes via padding), JIT-compiling once with those static shapes, and then replicating inputs/parameters across TPU chips for parallel image generation.
Where it helps
- Text-to-image serving with predictable prompt sizes and fixed output resolution
- Production inference on TPU where startup compilation can be amortized
- Batching multiple generations per request across TPU chips
- Systems that need stable latency after an initial warm-up
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
JIT compilation requires that input, intermediate, and output shapes remain static; changing prompt batching dimensions or image counts/sizes can trigger new costly compilations. Also, achieving good results may require careful handling of numeric precision choices (e.g., converting model parameters to bfloat16 while keeping scheduler state in float32, as described).
Sources
- 🧨 Accelerating Stable Diffusion XL Inference with JAX on Cloud TPU v5e ↗
Hugging Face · primary · Published 2023-10-03 · Checked 2026-11-10
