← AI Techniques Radar
Inference efficiency

Use JAX JIT with static shapes plus XLA parallelism (pmap) for SDXL TPU inference

Compile an SDXL image-generation pipeline in JAX with just-in-time (JIT) compilation that assumes fixed (static) input/output shapes, then scale generation across multiple TPU chips using pmap-style parallelism so multiple images render concurrently with the same compiled program.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

SDXL inference can be bottlenecked by compilation overhead and by how efficiently hardware can be used. This approach reduces repeated runtime work by compiling once for fixed shapes, and improves utilization by expressing single-program multiple-data parallel execution across TPU devices.

What changed

Instead of treating inference as fully dynamic per request, the pipeline is structured around constant tensor dimensions (including fixed prompt tensor shapes via padding), JIT-compiling once with those static shapes, and then replicating inputs/parameters across TPU chips for parallel image generation.

Where it helps

  • Text-to-image serving with predictable prompt sizes and fixed output resolution
  • Production inference on TPU where startup compilation can be amortized
  • Batching multiple generations per request across TPU chips
  • Systems that need stable latency after an initial warm-up

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

JIT compilation requires that input, intermediate, and output shapes remain static; changing prompt batching dimensions or image counts/sizes can trigger new costly compilations. Also, achieving good results may require careful handling of numeric precision choices (e.g., converting model parameters to bfloat16 while keeping scheduler state in float32, as described).

Sources