Instruction-conditioned image editing via instruction-tuned diffusion
Adapt a text-to-image diffusion model for image-to-image editing by instruction-tuning so it can take an input image plus an explicit instruction phrase and produce the edited result that matches the transformation specified in the prompt (e.g., cartoonization or deraining).
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
The approach aims to bridge a gap between natural-language editing goals and the concrete edits produced by image editing systems, by training the model to reliably follow task-specific instructions over image inputs rather than relying only on general editing behavior.
What changed
Instead of using a pre-trained InstructPix2Pix model as-is (which the authors report can struggle with more specific transformation instructions), the method trains an instruction-tuned model on instruction-image exemplar data for target tasks, using instruction-templated datasets (including synonym-based instruction generation for cartoonization).
Where it helps
- Instruction-based cartoonization from an input image
- Prompted image restoration such as deraining
- Prompted denoising or deblurring workflows
- Low-level image enhancement with task phrases
- Building a single instruction-following image editor across multiple tasks
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The authors report that results can be task- and data-dependent: some low-level tasks (such as low-light enhancement and, to some extent, deblurring) were described as leaving room for improvement, and the model may fail on classes not sufficiently represented during training; additionally, the authors note open questions about scaling datasets, training duration, and broadening the task mixture.
Sources
- Instruction-tuning Stable Diffusion with InstructPix2Pix ↗
Hugging Face · primary · Published 2023-05-23 · Checked 2026-11-10
