Driving · 2026-10-09

Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu, Wei Yang Bryan Lim

arXiv:2610.11942 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

For GUI agents, safety depends on whether the outcome matches the expected transition, not on whether the resulting screen looks legitimate. The authors argue that verification should predict the next observation representation directly (decoder-free) and compare vectors. They present LGWM, trained on 1.85M real GUI transitions, and evaluate on RSWT-Bench where outcome-only detectors are at chance.

Why it matters

The paper targets GUI-agent attacks where a hijacked transition yields a pixel-identical “legitimate” screen. It proposes a verifier that checks transition integrity by vector comparison in representation space, avoiding a second judge model needed when predictions are rendered as text/images/code. Reported results show strong AUC and low per-decision latency on a donor-paired benchmark designed to make screen-only detection ineffective.

Method

LGWM is an action-conditioned latent GUI world model: from a current screenshot and structured action, it predicts the next screen’s representation directly, with no decoder and no judge at inference.
Training is self-supervised on 1.85M real GUI transitions, with a change-weighted patch loss and no semantic annotation.
Verification uses residual readouts: a training-free integrity (surprise) score from representation alignment gap, and a supervised linear head using the residual direction to separate harmful from benign violations.

Limitation

The authors state that “the adversary’s problem” becomes forging an outcome in the agent’s own representation space and that “the difficulty we have not yet quantified.”

Abstract (from arXiv)

A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions. Verification reduces to a vector comparison, and the same signal reveals whether a mismatch is harmful. We evaluate on RSWT-BENCH, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction. The training-free score reaches 0.987 AUC at 17 ms per decision, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual direction reaches 0.953 AUC at separating harmful from benign violations, where prompted VLMs are near chance. Further analyses show that the prediction is a usable future state rather than an anomaly score. World models have mostly served as simulators or planners; our results point to a third role, verification, for which predicting in representation space is the natural design.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-09 UTC

preprint