Pub-AI: AI in Science

Driving · 2026-09-30

The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models

Dongsheng Liu, Chao Jin, Wenkui Yang, Hejin Wang, Junwei Yang, Zeren Zhang, Ziwei Chen, Huaibo Huang, Jie Cao, Ran He

arXiv:2609.32679PDF

Auto-summarized: this summary was generated by a language model from the paper’s text and has not been reviewed by an editor. Check the paper before relying on it. Benchmark numbers appear only after a human has verified them.

TL;DR

GUI world models often predict future interfaces using only the current GUI observation and action, which can cause state aliasing: identical visible conditions correspond to different valid futures due to hidden transition-relevant state. The authors introduce StateAliasBench (strict-pair diagnostic) and a predictive-state recovery method that infers structured state from history and condition frozen GUI world models to restore state-sensitive prediction and improve AndroidWorld agent performance.

Why it matters

The paper argues that observation-only conditioning can systematically fail to disambiguate futures when the GUI omits transition-relevant state. By isolating this failure mode with strict pairs and benchmark suites (persistent artifact, clipboard, ordered collection, temporal deadline), the authors show that adding recovered predictive state improves both state-sensitive future prediction and downstream GUI-agent success, without retraining the underlying world models.

Method

  • Introduce StateAliasBench: strict pairs built from real Android interaction branches that match the same model-visible observation and next action while differing in transition-relevant hidden state and yielding different validated semantic outcomes.
  • Define predictive-state recovery: train a unified estimator via multi-teacher distillation from family-specific state specialists to recover structured predictive state from interaction history.
  • Condition frozen GUI world models with a deterministic, model-compatible state-conditioning interface so that predictive-state augmentation improves state-sensitive prediction while preserving next-observation fidelity.

Limitation

The benchmark state aliasing is instantiated using four state families (Persistent Artifact, Clipboard, Ordered Collection, Temporal Deadline), which represent only a subset of latent state variables in general-purpose interactive systems; additionally, the structured state used is a family-specific approximation to the canonical predictive state, so residual representation error and frozen-world-model approximation error may remain, meaning the method does not imply complete recovery of the underlying transition law.

Abstract (from arXiv)

GUI World Models (GUI-WMs) are increasingly used to predict future states for agent planning and simulation, yet most existing formulations condition only on the current GUI observation and action. We identify state aliasing, where the vis- ible interface omits transition-relevant environment state, so identical observable conditions can correspond to different valid futures. To diagnose this failure mode, we introduce StateAliasBench, a diagnostic benchmark that explicitly isolates such ambiguities via strict pairing. We further propose lightweight predictive- state recovery that infers structured state from history and augments otherwise frozen GUI-WMs through a deterministic state interface. Family-specific special- ists provide state recovery across heterogeneous state types, and multi-teacher dis- tillation consolidates them into a single unified estimator. Experiments show that existing GUI-WMs exhibit systematic failures under observation-only condition- ing, while predictive-state augmentation substantially restores state-sensitive pre- diction across evaluated WMs, preserves generative fidelity, and improves down- stream performance of GUI agents on AndroidWorld. These results suggest that reliable GUI world modeling should account not only for what is visible, but also for the hidden transition state that determines what happens next.

Related papers