Video World Models · 2026-11-05

MKD2World: A Typed Spatial MarkDown World Model For Web Agents

Zhihao Nan, Sifan Zhou, Yiming Cheng, Jinkai Zheng, Zheng Zhu, Si Shi, Ming Li, Fei Yu

arXiv:2611.06544PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors introduce Typed Spatial Markdown (TSM), a structured UI state for desktop web interfaces that encodes UI element types, visual semantics, explicit bounding boxes, optional control states, and hierarchical membership. They build AtomBlock-WebUI to train a screenshot-to-TSM renderer. Building on TSM, MKD2World is a two-stage, action-conditioned world model that predicts next TSM states and supports digest-based action selection for web agents.

Why it matters

GUI agents need foresight that reflects the post-action interface, including layout and UI semantics needed to reason about action effects. The authors propose a compact, structured state representation (TSM) and a way to train it from screenshots with aligned type and location supervision (AtomBlock-WebUI), then use it for action-conditioned next-state prediction (MKD2World) and action selection.

Method

  • Introduce Typed Spatial Markdown (TSM): a Markdown-style UI state that serializes block-level and atomic-level elements with types, visual semantics, bounding boxes, optional control states, and hierarchical membership.
  • Create AtomBlock-WebUI (9.7k-scale) using injected element-type labels into HTML, rendered in Chromium, and direct bounding-box extraction from the corresponding yolo-* DOM nodes to avoid post-hoc alignment/type inference errors.
  • Train MKD2World in two stages: screenshot-to-current TSM reconstruction, then action-conditioned next-state prediction using transition-aware objectives and a digest-based inference strategy for action selection.

Limitation

The provided text does not state specific limitations of the method.

Abstract (from arXiv)

Graphical user interface (GUI) agents can improve decision-making with world-model foresight, provided that predicted states accurately reflect post-action interfaces. Existing state representations fail to jointly capture interface layout, UI semantics, and explicit locations. We introduce Typed Spatial Markdown (TSM), a Markdown-style textual state representation for desktop web interfaces. TSM serializes block-level and atomic-level UI elements according to their spatial relationships, jointly encoding their types, visual semantics, explicit locations, optional control states, and hierarchical membership. To train a screenshot-to-TSM renderer, we construct AtomBlock-WebUI, a dataset with precise annotations at both granularities. By injecting element-type labels into HTML and directly extracting bounding boxes after rendering, we avoid annotation errors caused by post-hoc inference from webpage structures. Building on TSM, we propose MKD2World, a desktop web world model trained in two stages for current-state description and action-conditioned next-state prediction. During inference, the web agent queries MKD2World as needed and performs a ``digest'' step, in which it interprets each predicted state and extracts task-relevant evidence for selecting the best action. Experiments show that MKD2World-9B achieves an EleF1 score of 26.50 on the out-of-domain test set, outperforming GPT-5.6-Sol(14.98) by 11.52 points. Across Online-Mind2Web tasks, MKD2World-9B improves the success rate of the Qwen3.5-35B web agent from 22.94% to 37.61%, achieving competitive performance among leading open-source GUI world models. Code is available at https://github.com/OmniAI-Lab-CUHKSZ/MKD2World.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-04 · Indexed by Pub-AI: 2026-11-05 UTC

preprint

MKD2World: A Typed Spatial MarkDown World Model For Web Agents

Zhihao Nan, Sifan Zhou, Yiming Cheng, Jinkai Zheng, Zheng Zhu, Si Shi, Ming Li, Fei Yu

Preprint posted: 2026-11-04 · Original record ↗

arxiv
2611.06544
Explicit source versions (1)
  • v1 · 2026-11-04
  1. Preprint available

    Source date: 2026-11-04 · First detected: 2026-11-05 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Graphical user interface (GUI) agents can improve decision-making with world-model foresight, provided that predicted states accurately reflect post-action interfaces. Existing state representations fail to jointly capture interface layout, UI semantics, and explicit locations. We introduce Typed Spatial Markdown (TSM), a Markdown-style textual state representation for desktop web interfaces. TSM serializes block-level and atomic-level UI elements according to their spatial relationships, jointly encoding their types, visual semantics, explicit locations, optional control states, and hierarchical membership. To train a screenshot-to-TSM renderer, we construct AtomBlock-WebUI, a dataset with precise annotations at both granularities. By injecting element-type labels into HTML and directly extracting bounding boxes after rendering, we avoid annotation errors caused by post-hoc inference from webpage structures. Building on TSM, we propose MKD2World, a desktop web world model trained in two stages for current-state description and action-conditioned next-state prediction. During inference, the web agent queries MKD2World as needed and performs a ``digest'' step, in which it interprets each predicted state and extracts task-relevant evidence for selecting the best action. Experiments show that MKD2World-9B achieves an EleF1 score of 26.50 on the out-of-domain test set, outperforming GPT-5.6-Sol(14.98) by 11.52 points. Across Online-Mind2Web tasks, MKD2World-9B improves the success rate of the Qwen3.5-35B web agent from 22.94% to 37.61%, achieving competitive performance among leading open-source GUI world models. Code is available at https://github.com/OmniAI-Lab-CUHKSZ/MKD2World.