← AI Techniques Radar
Training & fine-tuning

Task-prefix prompting for sequence-to-sequence vision-language fine-tuning

Fine-tune a vision-language model by framing each training example as a sequence-to-sequence generation problem where the text input is built from a task-specific prefix plus the question/prompt, paired with the corresponding image, and the target text is the desired answer/label.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

It provides a simple, repeatable way to adapt a general vision-language model to a downstream vision-language task when the base model does not natively support the desired I/O format or task token.

What changed

The approach changes the model’s training inputs by prepending a task identifier/prefix (e.g., a DocVQA tag) to each question, then training the model to generate the ground-truth answer text from the image and prefixed question; it also optionally adjusts fine-tuning scope by freezing or unfreezing the vision encoder.

Where it helps

  • Document visual question answering (DocVQA)
  • Adapting vision-language models to a new question/answer format
  • Instruction/prompt-to-text supervised fine-tuning for image-conditioned tasks
  • Reducing training cost by freezing the vision encoder during adaptation
  • Domain/task customization when a model lacks native support

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

A task must be expressible as text generation conditioned on the image and prefixed prompt; prompts that do not match the model’s expected conditioning format can lead to unusable outputs, and the text-based framing may not perfectly align with some evaluation/task conventions.

Sources