LLM-Assisted VQA Evaluation via Rationale-Based Answer Rating (LAVE)
Convert zero-shot VQA scoring into an LLM-driven answer-rating task: provide a question, a set of expert reference answers, and a candidate answer, then ask the LLM to output a single rating on a small discrete scale along with a rationale. The final evaluation score is derived from the selected rating.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
In out-of-distribution (OOD) settings, exact-match string metrics can penalize answers that are semantically correct but differ in format, specificity, or phrasing. LAVE aims to better reflect correctness as judged against reference answers using an LLM as an evaluator.
What changed
Instead of using strict overlap-based VQA metrics (e.g., CIDEr/BLEU/ANLS), the evaluation is reframed as an LLM-assisted judgment problem where the model rates candidate answers relative to expert references using a prompted rubric.
Where it helps
- Zero-shot evaluation of vision-language models on synthetic document VQA datasets
- Out-of-distribution VQA where exact-match scoring is overly strict
- Task evaluation when references exist but answer formatting may vary
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The approach depends on the evaluator LLM complying with the prompt and on the reliability of mapping a short discrete rating to evaluation quality; the source also notes that additional ablation work is needed for a complete understanding of metric behavior.
Sources
- LAVE: Zero-shot VQA Evaluation on Docmatix with LLMs - Do We Still Need Fine-Tuning? ↗
Hugging Face · primary · Published 2024-07-25 · Checked 2026-11-10
