Two-stage language-specific fine-tuning for text-to-image models
Adapt a pretrained text-to-image diffusion model to a specific language via two-stage fine-tuning: first train a language-specific text encoder with the image generator fixed, then jointly fine-tune both the text encoder and latent diffusion model so the generator produces language-appropriate visuals from prompts.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
It addresses prompt-tokenization mismatch and concept/style drift when the base model was trained mainly on another language, improving language-appropriate interpretation and outputs as claimed by the authors.
What changed
Stage 1 trains a Japanese-specific text encoder while freezing the latent diffusion model and uses a Japanese tokenizer; Stage 2 jointly fine-tunes the text encoder and latent diffusion model to better produce Japanese-style outputs from Japanese prompts.
Where it helps
- Japanese text-to-image generation
- Adapting prompt understanding to language-specific vocabulary and onomatopoeia
- Improving cultural/identity expression that may not transfer well via translation
- Fine-tuning image generation models for other low-resource languages with smaller caption datasets
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The approach described is presented as language-specific and may be less versatile than the original baseline; accuracy issues remain.
Sources
- Japanese Stable Diffusion ↗
Hugging Face · primary · Published 2022-10-05 · Checked 2026-11-10
