Driving · 2026-11-04
Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials
Anh Nguyen, Tianyi Xu, Phat Tran, Harry Yu, Huazheng Wang, Stefan Lee
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study how much world-model look-ahead helps computer-use agents when they are already conditioned on task-specific tutorials. They introduce OSWorld-Tutorial and ScienceBoard-Tutorial and propose RAG-PolicyJudge (RAG-PJ). With tutorials present, RAG-PJ gains little benefit from either zero-shot text rollouts or oracle visual rollouts, even with perfect future screenshots; the bottleneck is the judge’s reasoning over futures and recognizing task completion.
Why it matters
The paper isolates the contribution of world-model rollouts in tutorial-conditioned computer-use agents. The authors show that when tutorials are available, improvements from rollout-based look-ahead are small, shifting attention from rollout fidelity to the judge’s ability to reason over predicted futures and decide task success.
Method
- Construct OSWorld-Tutorial and ScienceBoard-Tutorial from OSWorld-Verified and ScienceBoard with curated multimodal tutorials.
- Propose RAG-PolicyJudge: a tutorial-conditioned policy proposes candidate actions and a tutorial-conditioned judge selects among them.
- Ablate world-model rollouts by comparing RAG-PJ with no rollout, zero-shot text-based next-state descriptions, and oracle visual futures (ground-truth screenshots).
Abstract (from arXiv)
Recent work has improved computer-use agents (CUAs) by combining retrieved task-relevant tutorials and look-ahead planning using world models with MLLM judges. However, the contribution of each component -- tutorials, the judge, and the world model -- to overall improvement has not been evaluated in isolation. To fill this gap, we first construct OSWorld-Tutorial and ScienceBoard-Tutorial: two datasets for CUAs with curated multimodal tutorials, covering everyday and scientific domains. We then propose RAG-PolicyJudge (RAG-PJ), a policy-judge framework for CUAs conditioned on task-specific tutorials: the policy proposes candidate actions, and the judge selects among them. To isolate the effect of world models given tutorials, we compare RAG-PJ with and without common world models: using either zero-shot text-based descriptions of next states or an oracle that returns ground-truth visual observations. We find that RAG-PJ gains little benefit from these rollouts when tutorials are present, even with perfect visual futures. This suggests prior world-model gains in tutorial-conditioned methods come mainly from tutorial understanding and the judge's ability to select the proposed actions, rather than from predicted rollouts. Error analysis reveals that the bottleneck is not rollout fidelity, but the judge's ability to reason over future states and recognize task completion. This highlights the need to improve not just next-state prediction in world models, but also reasoning over their predicted rollouts. Datasets and code are available at https://github.com/realjoenguyen/RAGPJ-official.
Related papers
- AnyGoal: On World Model Generalization Beyond Expert Rollouts
- Agent World Model in the Wild: Computer-Use Agent Environment Synthesis
- pico-WAM: Evaluating Counterfactual Prediction and Planning Error in a Compact World Action Model
- Learning to Act with World Models by Valuing What Comes Next
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-04 · Indexed by Pub-AI: 2026-11-04 UTC
Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials
First announced by arXiv: 2026-11-04 · Original record ↗
- arxiv
- 2611.04073
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-04 · First detected: 2026-11-04 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.04073v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Recent work has improved computer-use agents (CUAs) by combining retrieved task-relevant tutorials and look-ahead planning using world models with MLLM judges. However, the contribution of each component -- tutorials, the judge, and the world model -- to overall improvement has not been evaluated in isolation. To fill this gap, we first construct OSWorld-Tutorial and ScienceBoard-Tutorial: two datasets for CUAs with curated multimodal tutorials, covering everyday and scientific domains. We then propose RAG-PolicyJudge (RAG-PJ), a policy-judge framework for CUAs conditioned on task-specific tutorials: the policy proposes candidate actions, and the judge selects among them. To isolate the effect of world models given tutorials, we compare RAG-PJ with and without common world models: using either zero-shot text-based descriptions of next states or an oracle that returns ground-truth visual observations. We find that RAG-PJ gains little benefit from these rollouts when tutorials are present, even with perfect visual futures. This suggests prior world-model gains in tutorial-conditioned methods come mainly from tutorial understanding and the judge's ability to select the proposed actions, rather than from predicted rollouts. Error analysis reveals that the bottleneck is not rollout fidelity, but the judge's ability to reason over future states and recognize task completion. This highlights the need to improve not just next-state prediction in world models, but also reasoning over their predicted rollouts. Datasets and code are available at https://github.com/realjoenguyen/RAGPJ-official.
