Driving · 2026-10-02
AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
Cheng Yang, Yifan Wu, Yutao Huang, Zhaohua Zhang, Beiduo Chen, Muxi Chen, Chenchen Zhao, Hexuan Deng, Haolin Yang, Geyuan Zhu, Sa Zhu, Jianhuan Zhuo, Qiuyong Xiao, Jianhao Ruan, Yiran Peng, Jiayi Zhang, Tian Ye, Xinlei Yu, Tianwen Jiang, Jihong Zhang, Yuyu Luo
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AutoGUIWorld generates GUI interaction trajectories without deploying or running the underlying software. It samples initial GUI scenes from OS/visual/interface specifications, uses a planner to specify atomic actions and intended visual consequences, and iteratively edits screenshots with an image generator. It yields 79,266 grounded, filtered step-level samples across Ubuntu, Windows, macOS, and Chrome, and improves GUI-agent fine-tuning on OSWorld and ScienceBoard.
Why it matters
The authors address the cost and coverage limits of collecting GUI-agent training data by synthesizing action-grounded visual trajectories. They claim that planner-guided screenshot editing can provide supervision for GUI agents without executing the target software environments, and they report improved performance on real desktop/scientific benchmarks via fine-tuning.
Method
- Sample initial GUI seeds from structured OS substrate, visual appearance, and interface-state specs; realize seeds as screenshots with metadata.
- Generate task instructions conditioned on the fixed seed; a meta planner specifies atomic actions and intended visual consequences per step.
- Use Voyager and Image2 to edit the current screenshot into the next screenshot; ground pointing actions via LocateAnything and filter with action/transition quality checks.
Abstract (from arXiv)
GUI agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step workflows. However, the diversity of available trajectories is constrained by the applications, interface states, and workflows accessible in the underlying environments. Expanding this coverage requires deploying increasingly diverse and complex software, with specialized applications imposing additional installation, configuration, and runtime costs. We introduce AutoGUIWorld, a data generation framework that combines the visual priors of image generators with the task knowledge of a planner to synthesize GUI interaction trajectories without deploying or running the corresponding software environments. AutoGUIWorld samples initial GUI scenes from structured specifications of operating-system context, visual appearance, and interface state, and generates tasks conditioned on those scenes. A planner then specifies atomic actions and their intended visual consequences, while an image generator iteratively edits the current screenshot to produce subsequent observations. Action grounding and transition-level quality filtering yield 79,266 spatially annotated step-level training samples across Ubuntu, Windows, macOS, and Chrome. Fine-tuning Qwen3.5-35B-A3B on AutoGUIWorld trajectories improves the mean task score on OSWorld from 33.0% to 40.8% and the task success rate on ScienceBoard from 14.0% to 32.2%. These results show that generated trajectories improve GUI-agent performance on real desktop and scientific tasks.
Related papers
- Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training
- Latent World Action Model for GUI Agents
- The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models
- Beyond Next-Screen Prediction: The Decision Value of GUI World Models
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC
AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
Preprint posted: 2026-10-01 · Original record ↗
- arxiv
- 2610.01215
Explicit source versions (1)
- v1 · 2026-10-01
- Preprint available
Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
GUI agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step workflows. However, the diversity of available trajectories is constrained by the applications, interface states, and workflows accessible in the underlying environments. Expanding this coverage requires deploying increasingly diverse and complex software, with specialized applications imposing additional installation, configuration, and runtime costs. We introduce AutoGUIWorld, a data generation framework that combines the visual priors of image generators with the task knowledge of a planner to synthesize GUI interaction trajectories without deploying or running the corresponding software environments. AutoGUIWorld samples initial GUI scenes from structured specifications of operating-system context, visual appearance, and interface state, and generates tasks conditioned on those scenes. A planner then specifies atomic actions and their intended visual consequences, while an image generator iteratively edits the current screenshot to produce subsequent observations. Action grounding and transition-level quality filtering yield 79,266 spatially annotated step-level training samples across Ubuntu, Windows, macOS, and Chrome. Fine-tuning Qwen3.5-35B-A3B on AutoGUIWorld trajectories improves the mean task score on OSWorld from 33.0% to 40.8% and the task success rate on ScienceBoard from 14.0% to 32.2%. These results show that generated trajectories improve GUI-agent performance on real desktop and scientific tasks.