Training & fine-tuning
Adapt a pretrained text-to-image diffusion model to a specific language via two-stage fine-tuning: first train a language-specific text encoder with the image generator fixed, then jointly fine-tune both the text encoder and latent diffusion model so the generator produces language-appropriate visuals from prompts.
AI summary
Inference efficiency
Run the same diffusion pipeline in an SPMD fashion across multiple JAX devices by (1) replicating model parameters to all devices, (2) sharding tokenized prompt inputs so each device processes its own slice, and (3) using JAX parallel primitives (e.g., pmapping the pipeline’s internal generate step) so each device produces separate images in parallel.
AI summary
Training & fine-tuning
When fine-tuning Stable Diffusion with Dreambooth, add a prior-preservation step that includes images from the target’s broader class (e.g., a generic “person” class when training a new face). During fine-tuning, these class images are intended to counterbalance the new concept’s tendency to overfit.
AI summary
Robotics & embodied AI
Back to Articles Diffusion Models Live Event Published November 25, 2022 Update on GitHub Upvote 4 Lewis Tunstall
Source excerpt
Inference efficiency
Use a diffusion model whose forward noising and reverse denoising operate in a discrete latent token space. Images are first encoded into quantized token indices via a VQ-VAE/VQGAN-style encoder with a fixed codebook, and then a conditional encoder-decoder transformer learns to predict the categorical distribution of the un-noised latent tokens conditioned on a text prompt.
AI summary
Inference efficiency
Use Apple-provided converted Stable Diffusion Core ML checkpoints, choose between attention implementation variants, and control which compute units (CPU, GPU, Neural Engine) Core ML is allowed to use during inference.
AI summary
Agents & tool use
Use a pure-Elixir implementation of Hugging Face Transformers (Bumblebee) to integrate pre-trained transformer models into an Elixir application, leveraging the surrounding tooling (e.g., Livebook smart cells) to scaffold common ML workflows.
AI summary
Training & fine-tuning
Train a reward (preference) model on human-labeled rankings of generated text, then fine-tune a pretrained language model with reinforcement learning using Proximal Policy Optimization (PPO). The RL objective uses the reward model’s output combined with a constraint term that penalizes divergence from the original model via a KL-based penalty, to avoid destabilizing generation and reward-model exploitation.
AI summary
Training & fine-tuning
Instruction Fine-Tuning adapts a base language model by fine-tuning it on demonstrations that pair user instructions with corresponding inputs (optional) and desired outputs, using a diverse set of instruction-following tasks to teach helpful compliance with user-specified directions.
AI summary
Training & fine-tuning
Freeze the pretrained Stable Diffusion weights and train small low-rank adapter matrices (LoRA) inserted into the model’s cross-attention layers that connect image features to prompt text representations. After training, load the LoRA weights on top of the original base pipeline weights for inference.
AI summary
Agents & tool use
Back to Articles Introducing ⚔️ AI vs. AI ⚔️ a deep reinforcement learning multi-agents competition system Published February
Source excerpt
Inference efficiency
Use a Core ML–based diffusion pipeline that can automatically split computation across available hardware accelerators (CPU, GPU, Neural Engine) and select a best accelerator configuration for the current Mac. The approach is described as converting diffusion models to Core ML and relying on Core ML to decide which model portions run on which devices; the app also includes an automatic accelerator selection heuristic.
AI summary