← AI Techniques Radar
Inference efficiency

SPMD prompt replication and sharded inputs for JAX/Flax inference

Run the same diffusion pipeline in an SPMD fashion across multiple JAX devices by (1) replicating model parameters to all devices, (2) sharding tokenized prompt inputs so each device processes its own slice, and (3) using JAX parallel primitives (e.g., pmapping the pipeline’s internal generate step) so each device produces separate images in parallel.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

The approach aligns the pipeline’s data layout with JAX’s multi-device execution model, enabling parallel image generation while keeping the pipeline logic largely unchanged (each device runs the same computation on different input data).

What changed

Instead of sending a single prompt to a single device, the workflow prepares prompt token IDs for multiple prompts, replicates pipeline parameters across devices, shards the prompt IDs so each device receives its corresponding batch, and then executes a pmapped (SPMD) generation function (or uses the pipeline call with parallelism enabled).

Where it helps

  • Generating multiple images from one prompt across multiple devices
  • Generating images for multiple different prompts simultaneously
  • Running Flax diffusion inference on TPU-backed JAX environments
  • Leveraging sharded batches for multi-device evaluation

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

Inputs and array shapes generally need to be consistent across calls so JAX can reuse compiled computation; otherwise additional recompilation may occur.

Sources