Driving · 2026-10-20

Beyond Next-Screen Prediction: The Decision Value of GUI World Models

Sizhe Tang, Hanhan Zhou, Jeffrey Svajlenko, Rashmi Gangadharaiah

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors study how much GUI world-model predictions matter for action selection versus the evaluator that interprets those predictions. They show lookahead gains can change when predictions are presented to the evaluator, so end-to-end gains cannot be attributed to prediction alone. They introduce Direct Action Ranking (DAR), which scores candidates directly without generating future screens, and find predicted successors add no detectable ranking signal once evaluation is learned directly.

Why it matters

This work argues for evaluating GUI world models by their decision value: the incremental action-selection improvement attributable to predicted observations beyond a strong direct evaluator on the same candidates. It provides a matched-training protocol to separate prediction effects from evaluation effects and shows that a direct evaluator (DAR) can outperform world-model lookahead online while transferring zero-shot to unseen apps.

Method

Separates prediction from evaluation by holding the candidate actions and cached world-model predictions fixed while varying how an evaluator interprets them (Section 4.1).
Introduces Direct Action Ranking (DAR), a reference-supervised candidate scorer that selects actions without generating future screens (Section 5.1).
Measures decision value with matched supervision: compares DAR vs a prediction-augmented DAR where the only additional input is the predicted successor (Section 5.3).

Limitation

The authors note that the matched comparison bounds only the offline ranking signal of predictions; pooled AUROC and online success can diverge because execution depends on which within-set actions are selected.

Abstract (from arXiv)

GUI world models are increasingly used to improve action selection by predicting the consequences of candidate actions, but it is unclear how much of the resulting gain comes from prediction itself rather than from the evaluator that interprets those predictions. We study this attribution problem through controlled interventions on GUI action selection. With candidates and predictions held fixed, lookahead gains can change with how the prediction is presented to the evaluator, so end-to-end gains cannot be attributed to prediction alone. We then introduce Direct Action Ranking (DAR), which learns from reference actions to score candidates without generating future screens, and compare it with a prediction-augmented variant under matched supervision. Predicted successors add no detectable ranking signal once evaluation is learned directly. Without retraining, DAR improves AndroidWorld task success by 5.8--16.4 points across three proposers, outperforms world-model lookahead, and transfers zero-shot to unseen apps, where every prediction-conditioned configuration falls below proposer confidence. These results suggest that GUI world models should be evaluated by their decision value, the incremental action-selection value of their predictions beyond a strong direct evaluator, rather than by prediction fidelity or end-to-end lookahead gain alone.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-17 · Indexed by Pub-AI: 2026-10-20 UTC

preprint