Driving · 2026-10-27

InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

Qinsi Wang, Wei Wen, Chong Zhou, Saksham Suri, Zechun Liu, Lemeng Wu, Hai Helen Li, Yiran Chen, Raghuraman Krishnamoorthi, Qi Qian

arXiv:2610.33966PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

InteractiveWorld is a verifiable, human-centric simulator for closed-loop interaction between an assistant and a human who executes the assistant’s natural-language guidance. It introduces InteractiveBench, a dynamic benchmark with 120 multi-turn egocentric tasks across six categories, scored by verifiable state-based rewards. Experiments with 12 frontier MLLMs show large gaps: Claude Fable 5.1 reaches 41.7% average success, with only 20.0% on Guidance and 25.0% on Planning.

Why it matters

The authors address the challenge that real-world assistance is dynamic and closed-loop, while existing benchmarks are often static and cannot test whether guidance changes subsequent outcomes. InteractiveWorld/InteractiveBench aim to evaluate helpfulness via multi-turn human-assistant collaboration where task success is verified from final environment states, enabling research focused on interactive MLLM assistants for physical-world tasks.

Method

  • Propose InteractiveWorld: a verifiable closed-loop human–assistant environment for egocentric assistants, implemented on Isaac Sim, with physical, human action, and semantic layers.
  • Introduce InteractiveBench: 120 human-reviewed, multi-turn tasks across six categories; assistant guidance shapes outcomes, and success is verified from predefined environment states via state-verifiable rewards.
  • Evaluate 12 frontier MLLMs as the Assistant agent using a fixed Human agent (Gemini 3.1 Pro) and score task pass rates and capability checks across defined state conditions.

Limitation

The authors note that executor choice can affect individual scores (pass count changes by at most three tasks out of 36, or 8.3 percentage points), and they do not claim the complete ranking is invariant across executors.

Abstract (from arXiv)

People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today's MLLMs truly help people in physical world? To answer this question, we propose InteractiveWorld, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduce InteractiveBench, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-25 · Indexed by Pub-AI: 2026-10-27 UTC

preprint

InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

Qinsi Wang, Wei Wen, Chong Zhou, Saksham Suri, Zechun Liu, Lemeng Wu, Hai Helen Li, Yiran Chen, Raghuraman Krishnamoorthi, Qi Qian

Preprint posted: 2026-10-25 · Original record ↗

arxiv
2610.33966
Explicit source versions (1)
  • v1 · 2026-10-25
  1. Preprint available

    Source date: 2026-10-25 · First detected: 2026-11-12 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today's MLLMs truly help people in physical world? To answer this question, we propose InteractiveWorld, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduce InteractiveBench, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.