Robotics · 2026-10-20
Agent World Model in the Wild: Computer-Use Agent Environment Synthesis
Zhaoyang Wang, Qianhui Wu, Haorui Wang, Simon Yu, Baolin Peng, Canwen Xu, Hao Cheng, Wenlin Yao, Boyi Liu, Yite Wang, Jianfeng Gao, Huaxiu Yao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
WildAWM is a scalable pipeline for synthesizing executable personal-computer environments and cross-interface tasks for training general computer-use agents (CUAs). It supports GUI, model context protocol (MCP), and CLI together, uses only open-source models for synthesis, and applies execution-based quality gates (heterogeneous rollouts, judge consistency, and counterfactual probes) to ensure solvability, difficulty, and verifier reliability. It synthesizes 341 environment–task pairs; training on these worlds improves generalization across 8 OOD benchmarks.
Why it matters
The authors target the challenge of scaling computer-use agent training without risky real-world interaction. WildAWM generates controllable, executable worlds spanning multiple interfaces on a single personal-computer workspace, and it focuses on task synthesis quality via execution-based validation and stress-testing verifier robustness. The paper also emphasizes reproducibility by using open-source models throughout synthesis and releasing the code and task corpus.
Method
- World synthesis: persona-driven, multi-day deterministic materialization of a personal-computer world with web applications, MCP-accessible services, and local files; uses WebHarbor mirror sites as a visual substrate.
- XAI task synthesis: generates cross-interface tasks requiring GUI+MCP+CLI; synthesizes a checklist-based executable verifier and rubric/hints; uses heterogeneous open-source agents and multi-level judgments during synthesis.
- Quality gates: five gates for judge consistency, initial unsatisfaction, task solvability, task difficulty (empirical), and a counterfactual probe to test judge-feedback robustness to incorrect outcome claims.
Abstract (from arXiv)
Agent learning is shifting from learning from human data toward learning through interaction with the world. However, scaling such learning directly in real-world computer systems is challenging and can introduce substantial security risks. We therefore propose Agent World Model in the Wild (WildAWM), a scalable pipeline for synthesizing controllable environments and tasks for training general computer-use agents (CUAs). Whereas existing approaches typically specialize in a single agent interaction interface, we construct executable environments that jointly support graphical user interfaces, model context protocol services, and command-line interfaces, allowing agents to flexibly combine visual interaction, tool use, and code execution. WildAWM is entirely powered by open-source models, demonstrating that high-quality computer-use environments can be generated without proprietary frontier models. Beyond environments, we address three recurring challenges in task synthesis: ensuring solvability, maintaining sufficient difficulty, and producing reliable verifiers, by introducing execution-based quality gates that combine heterogeneous agent rollouts, cross-judge consistency checks, and counterfactual probes of judgment robustness. The resulting pipeline synthesizes 341 long-horizon environment-task pairs that can require over 100 agent steps to solve and remain challenging even for frontier models, with GPT-6-Astra solving only 60% of them. We train two Qwen models of different sizes exclusively in these synthetic worlds. Both models achieve substantial gains across 8 out-of-distribution benchmarks spanning web, claw, and computer-use agents, demonstrating that learning in WildAWM's synthesized worlds generalizes broadly to unseen computer-use tasks. The code is publicly available at https://github.com/MSR-Orchard/WildAWM.
Related papers
- Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design
- Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks
- AgentGarten: Code Worlds for Evolving Agents
- Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards
- LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-20 UTC
Agent World Model in the Wild: Computer-Use Agent Environment Synthesis
Preprint posted: 2026-10-19 · Original record ↗
- arxiv
- 2610.26828
Explicit source versions (1)
- v1 · 2026-10-19
- Preprint available
Source date: 2026-10-19 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Agent learning is shifting from learning from human data toward learning through interaction with the world. However, scaling such learning directly in real-world computer systems is challenging and can introduce substantial security risks. We therefore propose Agent World Model in the Wild (WildAWM), a scalable pipeline for synthesizing controllable environments and tasks for training general computer-use agents (CUAs). Whereas existing approaches typically specialize in a single agent interaction interface, we construct executable environments that jointly support graphical user interfaces, model context protocol services, and command-line interfaces, allowing agents to flexibly combine visual interaction, tool use, and code execution. WildAWM is entirely powered by open-source models, demonstrating that high-quality computer-use environments can be generated without proprietary frontier models. Beyond environments, we address three recurring challenges in task synthesis: ensuring solvability, maintaining sufficient difficulty, and producing reliable verifiers, by introducing execution-based quality gates that combine heterogeneous agent rollouts, cross-judge consistency checks, and counterfactual probes of judgment robustness. The resulting pipeline synthesizes 341 long-horizon environment-task pairs that can require over 100 agent steps to solve and remain challenging even for frontier models, with GPT-6-Astra solving only 60% of them. We train two Qwen models of different sizes exclusively in these synthetic worlds. Both models achieve substantial gains across 8 out-of-distribution benchmarks spanning web, claw, and computer-use agents, demonstrating that learning in WildAWM's synthesized worlds generalizes broadly to unseen computer-use tasks. The code is publicly available at https://github.com/MSR-Orchard/WildAWM.