Video World Models · 2026-11-05
MKD2World: A Typed Spatial MarkDown World Model For Web Agents
Zhihao Nan, Sifan Zhou, Yiming Cheng, Jinkai Zheng, Zheng Zhu, Si Shi, Ming Li, Fei Yu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce Typed Spatial Markdown (TSM), a structured UI state for desktop web interfaces that encodes UI element types, visual semantics, explicit bounding boxes, optional control states, and hierarchical membership. They build AtomBlock-WebUI to train a screenshot-to-TSM renderer. Building on TSM, MKD2World is a two-stage, action-conditioned world model that predicts next TSM states and supports digest-based action selection for web agents.
Why it matters
GUI agents need foresight that reflects the post-action interface, including layout and UI semantics needed to reason about action effects. The authors propose a compact, structured state representation (TSM) and a way to train it from screenshots with aligned type and location supervision (AtomBlock-WebUI), then use it for action-conditioned next-state prediction (MKD2World) and action selection.
Method
- Introduce Typed Spatial Markdown (TSM): a Markdown-style UI state that serializes block-level and atomic-level elements with types, visual semantics, bounding boxes, optional control states, and hierarchical membership.
- Create AtomBlock-WebUI (9.7k-scale) using injected element-type labels into HTML, rendered in Chromium, and direct bounding-box extraction from the corresponding yolo-* DOM nodes to avoid post-hoc alignment/type inference errors.
- Train MKD2World in two stages: screenshot-to-current TSM reconstruction, then action-conditioned next-state prediction using transition-aware objectives and a digest-based inference strategy for action selection.
Limitation
The provided text does not state specific limitations of the method.
Abstract (from arXiv)
Graphical user interface (GUI) agents can improve decision-making with world-model foresight, provided that predicted states accurately reflect post-action interfaces. Existing state representations fail to jointly capture interface layout, UI semantics, and explicit locations. We introduce Typed Spatial Markdown (TSM), a Markdown-style textual state representation for desktop web interfaces. TSM serializes block-level and atomic-level UI elements according to their spatial relationships, jointly encoding their types, visual semantics, explicit locations, optional control states, and hierarchical membership. To train a screenshot-to-TSM renderer, we construct AtomBlock-WebUI, a dataset with precise annotations at both granularities. By injecting element-type labels into HTML and directly extracting bounding boxes after rendering, we avoid annotation errors caused by post-hoc inference from webpage structures. Building on TSM, we propose MKD2World, a desktop web world model trained in two stages for current-state description and action-conditioned next-state prediction. During inference, the web agent queries MKD2World as needed and performs a ``digest'' step, in which it interprets each predicted state and extracts task-relevant evidence for selecting the best action. Experiments show that MKD2World-9B achieves an EleF1 score of 26.50 on the out-of-domain test set, outperforming GPT-5.6-Sol(14.98) by 11.52 points. Across Online-Mind2Web tasks, MKD2World-9B improves the success rate of the Qwen3.5-35B web agent from 22.94% to 37.61%, achieving competitive performance among leading open-source GUI world models. Code is available at https://github.com/OmniAI-Lab-CUHKSZ/MKD2World.
Related papers
- Latent World Action Model for GUI Agents
- The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models
- AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
- Agent World Model in the Wild: Computer-Use Agent Environment Synthesis
- PrismWAM: Decoupling State Representations in World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-04 · Indexed by Pub-AI: 2026-11-05 UTC
MKD2World: A Typed Spatial MarkDown World Model For Web Agents
Preprint posted: 2026-11-04 · Original record ↗
- arxiv
- 2611.06544
Explicit source versions (1)
- v1 · 2026-11-04
- Preprint available
Source date: 2026-11-04 · First detected: 2026-11-05 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.06544v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Graphical user interface (GUI) agents can improve decision-making with world-model foresight, provided that predicted states accurately reflect post-action interfaces. Existing state representations fail to jointly capture interface layout, UI semantics, and explicit locations. We introduce Typed Spatial Markdown (TSM), a Markdown-style textual state representation for desktop web interfaces. TSM serializes block-level and atomic-level UI elements according to their spatial relationships, jointly encoding their types, visual semantics, explicit locations, optional control states, and hierarchical membership. To train a screenshot-to-TSM renderer, we construct AtomBlock-WebUI, a dataset with precise annotations at both granularities. By injecting element-type labels into HTML and directly extracting bounding boxes after rendering, we avoid annotation errors caused by post-hoc inference from webpage structures. Building on TSM, we propose MKD2World, a desktop web world model trained in two stages for current-state description and action-conditioned next-state prediction. During inference, the web agent queries MKD2World as needed and performs a ``digest'' step, in which it interprets each predicted state and extracts task-relevant evidence for selecting the best action. Experiments show that MKD2World-9B achieves an EleF1 score of 26.50 on the out-of-domain test set, outperforming GPT-5.6-Sol(14.98) by 11.52 points. Across Online-Mind2Web tasks, MKD2World-9B improves the success rate of the Qwen3.5-35B web agent from 22.94% to 37.61%, achieving competitive performance among leading open-source GUI world models. Code is available at https://github.com/OmniAI-Lab-CUHKSZ/MKD2World.
