Driving · 2026-11-09

AtariBench: Evaluating Spatiotemporal Reasoning of Vision-Language Models in Atari Gameplay

Mingjia Huo, Yao Fu, Bo Chang, Yaqing Wang, Dawei Zhu, Dylan Zhang, Xinyang Yi, Lichan Hong, Ed H. Chi, Hao Zhang, Jiaxi Tang

arXiv:2611.08755PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AtariBench evaluates vision-language models as game-playing agents on 20 Atari 2600 games, requiring sequential decisions from high-temporal-resolution gameplay streams. The benchmark decouples in-game dynamics from wall-clock inference speed and tests spatiotemporal reasoning and grounding in the current observation. The authors report substantial limitations in state-of-the-art VLMs, including motion/direction inference failures, brittle sequential decision-making, and over-reliance on textual interaction history.

Why it matters

The authors argue that static or turn-based vision evaluation hides a capability gap for dynamic decision-making. AtariBench provides a controlled, reproducible way to test whether VLMs can perceive rapid visual changes, reason over spatiotemporal dynamics, and ground actions in the present observation. The paper also includes separability analysis and investigations of memory design and prompting strategies, aiming to identify which in-game signals reflect model competence.

Method

  • Benchmark setup: 20 Atari 2600 games; models make sequential decisions directly from high-temporal-resolution gameplay streams within ALE.
  • Controlled temporal interface: render at 210×160; execute actions at 10 Hz by repeating each chosen action for three frames; disable ALE sticky actions; fixed 30-second evaluation window with early reset after termination.
  • Agentic harness: structured memory (recent-history buffer with K=3 turns and reward-history buffer with R=3 non-zero reward clips by default) plus textual reward/life-loss feedback; parse model output into 1–10 actions with invalid outputs falling back to no-op.

Limitation

The authors note that the VLM-judge results are descriptive: with only five turns from one selected run per model–game pair, the results are not intended to rank models or estimate failure rates across all benchmark trajectories, and the judge attribution is model-assisted (judge can misread visual evidence; stated rationale need not fully explain actions).

Abstract (from arXiv)

Vision-Language Models (VLMs) have achieved strong performance on static visual understanding benchmarks, yet their ability to act in dynamic environments remains less well understood. We introduce AtariBench, a controlled benchmark that evaluates VLMs across 20 Atari 2600 games, requiring models to make sequential decisions from high-temporal-resolution gameplay streams. By decoupling fast in-game dynamics from wall-clock inference speed, AtariBench focuses on whether models can perceive rapid visual state changes, reason over spatiotemporal dynamics, and ground actions in the current observation. Our experiments reveal substantial limitations in state-of-the-art VLMs: models often fail to infer object motion and direction, exhibit brittle sequential decision-making, and over-rely on textual interaction history despite newly appended visual feedback. We further introduce a separability analysis to identify which in-game scores provide reliable signals of model competence, and use the benchmark to study how memory design and prompting strategies affect performance. Overall, AtariBench exposes a persistent gap between passive visual perception and active decision-making, providing a lightweight and rigorous testbed for developing VLMs that can reason and act in dynamic environments.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-09 · Indexed by Pub-AI: 2026-11-09 UTC

preprint

AtariBench: Evaluating Spatiotemporal Reasoning of Vision-Language Models in Atari Gameplay

Mingjia Huo, Yao Fu, Bo Chang, Yaqing Wang, Dawei Zhu, Dylan Zhang, Xinyang Yi, Lichan Hong, Ed H. Chi, Hao Zhang, Jiaxi Tang

First announced by arXiv: 2026-11-09 · Original record ↗

arxiv
2611.08755
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-09 · First detected: 2026-11-09 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.08755v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Vision-Language Models (VLMs) have achieved strong performance on static visual understanding benchmarks, yet their ability to act in dynamic environments remains less well understood. We introduce AtariBench, a controlled benchmark that evaluates VLMs across 20 Atari 2600 games, requiring models to make sequential decisions from high-temporal-resolution gameplay streams. By decoupling fast in-game dynamics from wall-clock inference speed, AtariBench focuses on whether models can perceive rapid visual state changes, reason over spatiotemporal dynamics, and ground actions in the current observation. Our experiments reveal substantial limitations in state-of-the-art VLMs: models often fail to infer object motion and direction, exhibit brittle sequential decision-making, and over-rely on textual interaction history despite newly appended visual feedback. We further introduce a separability analysis to identify which in-game scores provide reliable signals of model competence, and use the benchmark to study how memory design and prompting strategies affect performance. Overall, AtariBench exposes a persistent gap between passive visual perception and active decision-making, providing a lightweight and rigorous testbed for developing VLMs that can reason and act in dynamic environments.