Driving · 2026-10-27
InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants
Qinsi Wang, Wei Wen, Chong Zhou, Saksham Suri, Zechun Liu, Lemeng Wu, Hai Helen Li, Yiran Chen, Raghuraman Krishnamoorthi, Qi Qian
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
InteractiveWorld is a verifiable, human-centric simulator for closed-loop interaction between an assistant and a human who executes the assistant’s natural-language guidance. It introduces InteractiveBench, a dynamic benchmark with 120 multi-turn egocentric tasks across six categories, scored by verifiable state-based rewards. Experiments with 12 frontier MLLMs show large gaps: Claude Fable 5.1 reaches 41.7% average success, with only 20.0% on Guidance and 25.0% on Planning.
Why it matters
The authors address the challenge that real-world assistance is dynamic and closed-loop, while existing benchmarks are often static and cannot test whether guidance changes subsequent outcomes. InteractiveWorld/InteractiveBench aim to evaluate helpfulness via multi-turn human-assistant collaboration where task success is verified from final environment states, enabling research focused on interactive MLLM assistants for physical-world tasks.
Method
- Propose InteractiveWorld: a verifiable closed-loop human–assistant environment for egocentric assistants, implemented on Isaac Sim, with physical, human action, and semantic layers.
- Introduce InteractiveBench: 120 human-reviewed, multi-turn tasks across six categories; assistant guidance shapes outcomes, and success is verified from predefined environment states via state-verifiable rewards.
- Evaluate 12 frontier MLLMs as the Assistant agent using a fixed Human agent (Gemini 3.1 Pro) and score task pass rates and capability checks across defined state conditions.
Limitation
The authors note that executor choice can affect individual scores (pass count changes by at most three tasks out of 36, or 8.3 percentage points), and they do not claim the complete ranking is invariant across executors.
Abstract (from arXiv)
People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today's MLLMs truly help people in physical world? To answer this question, we propose InteractiveWorld, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduce InteractiveBench, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.
Related papers
- RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
- Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds
- Learning Interactive Real-World Simulators
- Open-HumanWAM: An Open Recipe for Scaling Human and Robot Experience for Embodied Foundation Models
- Ropedia Xperience 2: A Gold Standard for Scaling Physical AI Environments and Behaviors
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-25 · Indexed by Pub-AI: 2026-10-27 UTC
InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants
Preprint posted: 2026-10-25 · Original record ↗
- arxiv
- 2610.33966
Explicit source versions (1)
- v1 · 2026-10-25
- Preprint available
Source date: 2026-10-25 · First detected: 2026-11-12 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today's MLLMs truly help people in physical world? To answer this question, we propose InteractiveWorld, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduce InteractiveBench, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.
