Unassigned · 2026-10-23

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Advait Gupta, Sachin Konan, Tom Yin, Gabriel Sarch, Zhuang Liu

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AgentVQA is an offline multiple-choice benchmark for agentic visual understanding across five domains (Web Agents, Egocentric Videos, Robotics, Games, Spatial Planning). It is built from 1,000 manually verified questions adapted from 23 datasets. Across 22 VLMs, the best model reaches 58.5% accuracy. The authors report errors are mainly due to failures to ground and maintain accurate representations of evolving visual state.

Why it matters

The authors argue online agent evaluation is costly and challenging to scale, so they introduce an offline benchmark intended to support faster, reliable comparisons across agentic domains. They report that AgentVQA model rankings track those from an online evaluation (SpatialWorld) in the matched domain, while diverging from general VQA benchmarks—suggesting conventional benchmarks may not predict agentic competence.

Method

Construct AgentVQA via a five-stage human-centered pipeline: collect candidates from 23 datasets, standardize to multiple-choice (with hard negatives), use VLM-assisted screening, perform manual curation/repair, and verify exactly one correct answer with plausible hard negatives.
Evaluate 22 VLMs on 1,000 manually verified questions (200 per domain) using a standard prompt that asks for the single letter choice; sample 32 frames for models without native video input.
Analyze failure modes with an error taxonomy and large-scale annotation using Gemini 3.1 Pro, with manual verification on a subset of 100 error-mode assignments.

Limitation

The paper reports that the cross-domain rank agreement is weaker outside the matched domain: “The lower correlations for other AgentVQA domains suggest that model rankings depend on the capabilities being assessed, highlighting the limitations of a single dataset as a proxy for overall agentic performance.”

Abstract (from arXiv)

Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC

preprint