Video World Models · 2026-10-06

Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training

Yongxin Ning, Runliang Niu, Qianli Xing, Zhiyi Duan, Qingzu He, Pan Wang, Qi Wang

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Infinite-Dreamer is a simulation-free GUI data synthesis method using a pixel-level Image Editing World Model. It treats GUI transitions as image editing guided by VLM-extracted structured delta-text, then fine-tunes an image editor to generate next screenshots and multi-step imaginary trajectories. The synthesized data trains Infinite-Actor (Qwen3-VL fine-tuned only on synthesized data), improving AndroidWorld and MobileWorld task success.

Why it matters

The paper targets the bottleneck of collecting massive high-fidelity visual-action GUI trajectories. By generating pixel-level screenshot transitions via controllable image editing (conditioned on delta-text) instead of relying on text/HTML renderings or simulators, it aims to create scalable training data for GUI agents and improve downstream agent performance on mobile benchmarks.

Method

Extract structured delta-text (trigger, added/removed UI elements, optional next-step affordances) from real screenshot/action pairs using a VLM extractor.
Fine-tune an instruction-guided image-editing backbone to predict next screenshots from (current screenshot, delta-text), enabling single-frame augmentation and multi-step autoregressive imagination.
Fine-tune a Qwen3-VL GUI agent (Infinite-Actor) using only synthesized data; filter imagined steps using a VLM quality checker and keep steps with confidence c_t ≥ η.

Limitation

The paper discusses that multi-step rollout quality declines over extended horizons (e.g., progressive decline from 70.71 at 1 step to 48.84 at 10 steps) due to semantic specification shift and visual memory amplification; it also notes regressions in coordinate grounding (Box Acc) on AC-Curated and a -2.7 point regression on AndroidWorld Hard tasks when mixing synthetic with real demonstrations.

Abstract (from arXiv)

Graphical User Interface (GUI) agents have emerged as a promising paradigm for automating complex digital workflows across diverse applications. However, training highly capable and generalizable agents fundamentally relies on massive, high-fidelity visual-action trajectories, which are notoriously difficult to acquire. While human demonstrations are unscalable, existing GUI world models rely on text descriptions or HTML rendering, discarding crucial pixel-level visual details like icons and layout styles. To address this issue, we introduce Infinite-Dreamer, a simulation-free data synthesis method powered by a pixel-level Image Editing World Model. By conceptualizing GUI transitions as image editing tasks, we leverage Vision-Language Models (VLMs) to describe action-induced UI changes as structured delta-text. We then fine-tune an image editing backbone to controllably synthesize realistic screenshot transitions. We utilize this model to generate both single-frame visual robustness data and multi-step imaginary trajectories. To validate the effectiveness of our approach, we fine-tune the Qwen3-VL baseline solely on the synthesized data to obtain Infinite-Actor, and evaluate it on AndroidWorld, MobileWorld, and AndroidControl-Curated benchmarks. Infinite-Actor consistently outperforms the Qwen3-VL baselines across scales: Infinite-Actor-8B improves AndroidWorld Pass@1 by +4.45 and nearly doubles the MobileWorld Pass@3 success rate, while Infinite-Actor-2B improves Pass@1 by +9.05. Code is available at https://github.com/swaydy-n/Infinite-Dreamer.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-05 · Indexed by Pub-AI: 2026-10-06 UTC

preprint