Task-prefix prompting for sequence-to-sequence vision-language fine-tuning
Fine-tune a vision-language model by framing each training example as a sequence-to-sequence generation problem where the text input is built from a task-specific prefix plus the question/prompt, paired with the corresponding image, and the target text is the desired answer/label.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
It provides a simple, repeatable way to adapt a general vision-language model to a downstream vision-language task when the base model does not natively support the desired I/O format or task token.
What changed
The approach changes the model’s training inputs by prepending a task identifier/prefix (e.g., a DocVQA tag) to each question, then training the model to generate the ground-truth answer text from the image and prefixed question; it also optionally adjusts fine-tuning scope by freezing or unfreezing the vision encoder.
Where it helps
- Document visual question answering (DocVQA)
- Adapting vision-language models to a new question/answer format
- Instruction/prompt-to-text supervised fine-tuning for image-conditioned tasks
- Reducing training cost by freezing the vision encoder during adaptation
- Domain/task customization when a model lacks native support
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
A task must be expressible as text generation conditioned on the image and prefixed prompt; prompts that do not match the model’s expected conditioning format can lead to unusable outputs, and the text-based framing may not perfectly align with some evaluation/task conventions.
Sources
- Fine-tuning Florence-2 - Microsoft's Cutting-edge Vision Language Models ↗
Hugging Face · primary · Published 2024-06-24 · Checked 2026-11-10
