← AI Techniques Radar
Inference efficiency

LLM-Assisted VQA Evaluation via Rationale-Based Answer Rating (LAVE)

Convert zero-shot VQA scoring into an LLM-driven answer-rating task: provide a question, a set of expert reference answers, and a candidate answer, then ask the LLM to output a single rating on a small discrete scale along with a rationale. The final evaluation score is derived from the selected rating.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

In out-of-distribution (OOD) settings, exact-match string metrics can penalize answers that are semantically correct but differ in format, specificity, or phrasing. LAVE aims to better reflect correctness as judged against reference answers using an LLM as an evaluator.

What changed

Instead of using strict overlap-based VQA metrics (e.g., CIDEr/BLEU/ANLS), the evaluation is reframed as an LLM-assisted judgment problem where the model rates candidate answers relative to expert references using a prompted rubric.

Where it helps

  • Zero-shot evaluation of vision-language models on synthetic document VQA datasets
  • Out-of-distribution VQA where exact-match scoring is overly strict
  • Task evaluation when references exist but answer formatting may vary

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

The approach depends on the evaluator LLM complying with the prompt and on the reliability of mapping a short discrete rating to evaluation quality; the source also notes that additional ablation work is needed for a complete understanding of metric behavior.

Sources