Driving · 2026-10-28
Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents
Pranjal Aggarwal, Sean Welleck
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Gym-Anything-RT builds 100 visual minigames (and non-real-time counterparts) to test vision-language agents under real-time interaction. The same minigame can be run live, slowed, or paused to separate real-time delays from underlying task difficulty. The authors report large capability gaps: GPT-6 Astra (xHigh) is 9.0% pass in real time vs 75.0% when paused, and they publicly release the benchmark and generated minigames.
Why it matters
The authors argue existing interactive evaluations often pause between actions or mix multiple capabilities in fixed games, making it unclear whether failures are due to slowness or missing abilities. Gym-Anything-RT isolates real-time interaction by controlling timing while preserving the task, and it provides non-real-time counterparts for controlled comparison. The results suggest speed is a primary bottleneck for frontier models and that coding can partially offset poor real-time performance.
Method
- Construct real-time visual minigames as programs with internal state, controls, and a clock; compare live, slowed, and paused configurations for the same minigame.
- Use a creator-auditor pipeline (creator implements program + oracle; auditors inspect code/evidence and verify solvability and real-time requirement).
- Add non-real-time counterparts by minimally changing timing, and vary difficulty (5 levels) and interaction mode (simplified vs full), preserving controls and grader.
Abstract (from arXiv)
AI agents have made substantial progress in domains where they have a non-trivial amount of time to deliberate between actions, such as mathematics, software engineering, and performing tasks on desktop applications. It is less clear how such models fare in settings that demand real-time interaction, such as playing a video game or operating a physical device. In these settings, the environment state may evolve as the agent is reasoning, or even as the underlying model is performing a forward pass. To gain insight into the real-time interaction capabilities of vision-language agents, we introduce Gym-Anything-RT. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. Gym-Anything-RT uses an agentic pipeline to automatically generate such environments based on a taxonomy of six capabilities required for real-time interaction, including the ability to perceive the environment and act under an evolving environment state. A key feature of Gym-Anything-RT is that it allows for controlled comparison of real-time interaction capabilities: we can run the same minigame in real time, or pause or slow down the environment to relax the requirement for real-time actions. We use Gym-Anything-RT to construct 100 real-time minigames and evaluate AI agents on them. We find large capability gaps in real-time interaction. For example, GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time minigames, which rises to 75.0% when the environment is paused. We also identify intriguing behaviors by the most powerful frontier agents: for instance, when given the ability to write code, these agents write code that bypasses the need to perceive and act in real time. We publicly release Gym-Anything-RT and the generated minigames to spur further research on AI agents in real-time interactive settings.
Related papers
- AgentGarten: Code Worlds for Evolving Agents
- EmbodiedEvo: Self-improving Coding Agents for Physical Intelligence
- Gym-Anything for Robotics: Scaling physical simulations with digital agents
- AgentVQA: A Unified Benchmark for Agentic Visual Understanding
- PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC
Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents
Preprint posted: 2026-10-27 · Original record ↗
- arxiv
- 2610.37567
Explicit source versions (1)
- v1 · 2026-10-27
- Preprint available
Source date: 2026-10-27 · First detected: 2026-10-28 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.37567v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
AI agents have made substantial progress in domains where they have a non-trivial amount of time to deliberate between actions, such as mathematics, software engineering, and performing tasks on desktop applications. It is less clear how such models fare in settings that demand real-time interaction, such as playing a video game or operating a physical device. In these settings, the environment state may evolve as the agent is reasoning, or even as the underlying model is performing a forward pass. To gain insight into the real-time interaction capabilities of vision-language agents, we introduce Gym-Anything-RT. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. Gym-Anything-RT uses an agentic pipeline to automatically generate such environments based on a taxonomy of six capabilities required for real-time interaction, including the ability to perceive the environment and act under an evolving environment state. A key feature of Gym-Anything-RT is that it allows for controlled comparison of real-time interaction capabilities: we can run the same minigame in real time, or pause or slow down the environment to relax the requirement for real-time actions. We use Gym-Anything-RT to construct 100 real-time minigames and evaluate AI agents on them. We find large capability gaps in real-time interaction. For example, GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time minigames, which rises to 75.0% when the environment is paused. We also identify intriguing behaviors by the most powerful frontier agents: for instance, when given the ability to write code, these agents write code that bypasses the need to perceive and act in real time. We publicly release Gym-Anything-RT and the generated minigames to spur further research on AI agents in real-time interactive settings.