Driving · 2026-11-06

Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

Chenhao Zheng, Zixian Ma, Kai Li, Kanchana Ranasinghe, Jieyu Zhang, Artemis Panagopoulou, Weikai Huang, Jianxu Shangguan, Yuanyuan Tan, Silvio Savarese, Juan Carlos Niebles, Ranjay Krishna, Ran Xu, Honglu Zhou

arXiv:2611.08286PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors introduce Interactive Visual Gym, a framework for closed-loop evaluation/training of visual interaction models by jointly simulating executable visual task environments and persona-conditioned simulated users. They build IVG-Bench with 315 interaction scenarios across cooking, household manipulation, and mobile-interface domains. Evaluating state-of-the-art VLMs as visual interaction models shows the best model, Claude Opus 5.5, completes 53.5% of tasks in time limits.

Why it matters

The paper argues that offline replay benchmarks miss how assistant responses change user behavior and task outcomes. Interactive Visual Gym provides executable environments with human-communicable action spaces, scenario-level rollout with controlled out-of-plan events, and evaluation traces for task completion, error detection, and interaction quality. This setup lets the authors test response latency effects and demonstrate closed-loop training improvements in a proof-of-concept RL experiment.

Method

  • Interactive Visual Gym combines executable task environments (CookSim, adapted VirtualHome, and ScreenSim) with simulated human users that collaborate with the assistant and revise actions in response to guidance.
  • Simulated users are persona-conditioned (skill, trust in AI, execution speed, and interruption preference) and can trigger scheduled out-of-plan events (errors, questions, proactive requests, plan changes).
  • Evaluation includes in-time task success, error detection recall, and interaction-quality metrics, and accounts for assistant response latency by delivering responses after a computed tick delay.

Limitation

The authors report that the benchmark evaluation uses a sampled set of state-of-the-art VLMs and that the best-performing model shows a “substantial gap toward reliable interactive assistance,” but no explicit limitation is stated in the provided text beyond this observed performance gap.

Abstract (from arXiv)

Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Claude Opus 5.5, completes only 53.5% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-05 · Indexed by Pub-AI: 2026-11-06 UTC

preprint

Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

Chenhao Zheng, Zixian Ma, Kai Li, Kanchana Ranasinghe, Jieyu Zhang, Artemis Panagopoulou, Weikai Huang, Jianxu Shangguan, Yuanyuan Tan, Silvio Savarese, Juan Carlos Niebles, Ranjay Krishna, Ran Xu, Honglu Zhou

Preprint posted: 2026-11-05 · Original record ↗

arxiv
2611.08286
Explicit source versions (1)
  • v1 · 2026-11-05
  1. Preprint available

    Source date: 2026-11-05 · First detected: 2026-11-06 UTC · Historical baseline

    Source evidence (3)
Abstract · arxiv

Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Claude Opus 5.5, completes only 53.5% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.