Latent Dynamics · 2026-10-14
Latent World Action Model for GUI Agents
Seungjun Yu, Hojun Choi, Seojeong Park, Jaeyo Shin, Hyunjung Shim
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose a latent GUI world-action model that shifts GUI world modeling from full pixel reconstruction to action-induced future information in latent space. Using a LoRA-adapted LMM with separate action and transition heads plus an action-aware contrastive objective, the model supports action selection with a single forward pass (transition head removed at inference). They report up to 31.9% (offline) and 40.6% (online) relative success-rate gains over pixel-space world models, and up to 98.9% lower inference latency.
Why it matters
Render-and-re-perceive GUI world modeling is computationally expensive and can encourage an identity shortcut where predicted future screens preserve the current state instead of capturing action-induced changes. The paper reframes the objective to learn latent action consequences for action selection, aiming to improve success rates while reducing inference latency.
Method
- Train a LoRA-adapted LMM to score candidate GUI actions from shared latent representations (action head).
- Add a transition head to predict action-conditioned next-state latent embeddings from observed actions, avoiding explicit screen reconstruction.
- Mitigate the identity shortcut using an action-aware contrastive objective that contrasts predicted next-state latents against current-state latents and latents from other transitions (FIFO memory bank).
Limitation
The paper does not state an explicit limitation in the provided text.
Abstract (from arXiv)
Pixel-space world models have recently emerged as a promising approach for GUI agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key challenges in GUI environments: (L1) the high computational cost of pixel reconstruction and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive experiments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%. Our code is available at https://github.com/sjyu001/latent2world.
Related papers
- Beyond Next-Screen Prediction: The Decision Value of GUI World Models
- Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training
- PrismWAM: Decoupling State Representations in World Action Models
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
- AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-14 UTC
Latent World Action Model for GUI Agents
Preprint posted: 2026-10-09 · Original record ↗
- arxiv
- 2610.14228
Explicit source versions (1)
- v1 · 2026-10-09
- Preprint available
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Pixel-space world models have recently emerged as a promising approach for GUI agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key challenges in GUI environments: (L1) the high computational cost of pixel reconstruction and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive experiments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%. Our code is available at https://github.com/sjyu001/latent2world.