Driving · 2026-10-02

Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation

Chuyao Fu, Xiaowei Chi, Yuhan Rui, Yu-kai Wang, Zezhong Qian, Xiaojie Zhang, Yunfan Lou, Kevin Zhang, Kuangzhi Ge, Chak Wing Mak, Zhiyang Chen, Athena Zhuoming Zhong, Hongyang Chen, Haoran Li, Yike Guo, Sirui Han, Shanghang Zhang

arXiv:2610.00575 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Token-World models robot manipulation world dynamics directly in a compact state derived from vision-language model (VLM) visual tokens. It compresses high-dimensional policy-facing VLM tokens into a lower-dimensional token state (via an S-VAE), learns action-conditioned autoregressive transitions with a flow-matching DiT, and maps predicted states back only when the downstream policy needs them—avoiding intermediate RGB generation. The authors report improved long-horizon open-loop fidelity and closed-loop success-rate correlation, with lower simulation latency than recent baselines.

Why it matters

The paper targets a mismatch in common VLA simulation pipelines: RGB-based world models require an indirect interface (RGB prediction + re-encoding to policy tokens), even though downstream policies consume policy-facing VLM tokens. Token-World proposes a representation-aligned token-space simulator with compact latent states and action-conditioned rollout, and evaluates it on simulated and real-world manipulation benchmarks.

Method

Compress policy-facing VLM visual tokens z_t into compact per-token latent state c_t with an S-VAE (feature-dimension compression only; spatial layout preserved).
Learn action-conditioned autoregressive dynamics in compact token space using a spatiotemporal Transformer (factorized spatial/temporal attention) with recurrent temporal GRU memory; train with an x0-parameterized flow-matching objective and shortcut forcing.
During rollout, generate future compact states in token space; map predicted states back to policy-facing VLM features via a frozen decoder only for downstream control/evaluation (auxiliary RGB decoding used only for visualization).

Abstract (from arXiv)

A common approach to world-model simulation for vision-language-action (VLA) systems is to predict future RGB observations and then re-encode them into policy inputs, introducing an indirect interface between simulation and downstream policy execution. We instead investigate whether world dynamics can be modeled in a compact, policy-oriented state derived from VLM visual tokens. A key challenge is that raw VLM visual tokens are high-dimensional, making efficient and accurate autoregressive dynamics modeling challenging. To address this, we introduce Token-World, an action-conditioned world model that compresses VLM features into a compact token state, learns future dynamics in this reduced space, and maps predicted states back to the original policy-facing representation for downstream use. Across manipulation benchmarks, Token-World improves open-loop feature fidelity and policy-action consistency over recent world-model simulators, with slower degradation over long rollout horizons. In closed-loop evaluation, its simulated policy performance correlates more strongly with reference policy performance than Ctrl-World ($r=0.794$ vs.\ $0.583$), while requiring lower simulation latency. Ablations further show that compact-representation design and dimensionality substantially affect future-state prediction. Code will be available at https://chuyaofu.github.io/Token-World/.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC

preprint