Driving · 2026-10-28

GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

Bingchao Wang, Yongcheng Jing, Shunyu Liu, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

GUI-JEPA is a decoder-free mobile GUI agent policy built on a JEPA encoder pretrained on unlabeled GUI screenshots. It embeds the screenshot once, uses an action- and region-conditioned latent predictor to form one latent per candidate action, and scores all candidates in a single batched pass (no token decoding). On AndroidControl, it is 2.8–4.0× faster than a same-protocol 7B VLM baseline and reaches 86–92% of its in-distribution accuracy with 6% of its parameters.

Why it matters

The paper targets latency and memory bottlenecks in mobile GUI agents that rely on 7B+ vision-language models decoding action tokens or keeping an LLM/VLM in the loop for candidate scoring. By moving action selection into the latent space of a frozen JEPA encoder and batching candidate scoring, GUI-JEPA removes the decoder from the decision path and improves speed/memory while maintaining strong in-distribution accuracy.

Method

Freeze a JEPA ViT-L/16 encoder pretrained with a JEPA objective on unlabelled GUI screenshots; cache patch tokens so policy training skips the backbone.
Enumerate a11y-tree actions/arguments into candidates; transform the screen latent once per candidate using a residual action- and ROI-conditioned latent predictor (39.9M), producing one candidate-conditioned latent per action.
Score each candidate-conditioned latent with a structural head (cosine-based latent score) plus a MiniLM text matcher; add span and click rankers; select argmax. No token or pixel is generated for actions.

Abstract (from arXiv)

Mobile GUI agents are typically fine-tuned 7B-scale vision-language models (VLMs) that generate each action as text. However, these VLMs decode every action token by token, and even agents that score candidate actions keep a large language model or VLM in the loop, making them slow and memory-hungry. To address these issues, we propose GUI-JEPA, to our knowledge the first decoder-free GUI agent policy built on a Joint-Embedding Predictive Architecture (JEPA), which selects actions in the latent space of a JEPA encoder pretrained on unlabelled GUI screenshots. Its action- and region-conditioned latent predictor turns the screen latent into one latent per candidate action, letting lightweight heads score all candidates against the instruction in a single batched pass, without decoding a token. We further show that the scorer relies on this latent pathway and that the predictor is trained for the decision itself: a randomly initialised predictor does just as well as one pretrained on screen dynamics. On AndroidControl, GUI-JEPA is 2.8-4.0x faster on the same GPU than a Qwen2.5-VL-7B agent we fine-tune with the same inputs, while reaching 86-92% of its in-distribution accuracy with 6% of its parameters. Code is available at https://github.com/bcwang-sjtu/GUI-JEPA

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC

preprint