Video World Models · 2026-10-14
Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence
Thinker Team, Peng Gao, Jichao Jiao, Chen Liao, Daqin Luo, Yiming Ma, Baiyu Pan, Hailin Shi, Jiyuan Wang, Xiangming Xu, Xiongwei Zhao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Thinker-WM is a unified multimodal world model that jointly models and generates RGB video, metric depth, and robot action using a shared Unified Dynamics Transformer with modality-specific encoders/decoders. The authors unify at the representation level so appearance, geometry, and action tokens co-evolve in a common dynamic representation under task-level conditioning. The same parameterization supports multiple world-action functions.
Why it matters
World models often treat video, geometry, and executable actions as separate components. Thinker-WM instead unifies multimodal evolution inside a shared dynamics backbone, aiming to make metric structure a native part of the generative world state while supporting several embodied functions (e.g., joint video–action generation and inverse dynamics) from one model.
Method
- Representation-level unification: modality-specific encoders map RGB, depth, and action into a shared intermediate dynamic representation; a shared Unified Dynamics Transformer models their joint evolution; modality-specific decoders generate each modality.
- Geometry as a native generative modality: depth is represented using the same latent and spatiotemporal organization as visual generation, and depth supervision shapes the shared representation.
- Unified parameterization for multiple functions under task-level conditioning, including joint video–action generation, world prediction, policy inference, action-conditioned generation, video-to-action generation, and inverse dynamics.
Limitation
Although Thinker-WM achieves promising closed-loop control in simulation (e.g., RoboTwin), its effectiveness as a closed-loop policy on real robots remains to be established.
Abstract (from arXiv)
World models have emerged as a promising paradigm for embodied AI, yet existing systems often remain fragmented in the modalities they model and the functions they support. In particular, visual dynamics, geometric structure, and executable action are commonly coupled through separate interfaces or downstream modules rather than being jointly represented and evolved within a shared dynamics model. In this work, we present Thinker-WM, a unified multimodal world model that jointly models and generates RGB, metric depth, and robot action through a shared intermediate Unified Dynamics Transformer with modality-specific encoders and decoders. The model performs multimodal unification at the representation level: appearance, geometry, and action are transformed into a common dynamic representation and jointly evolved by shared parameters, while their distinct output structures are preserved. The same parameterization supports multiple functions, including joint video-action generation, world prediction, policy inference, action-conditioned generation, video-to-action generation, and inverse dynamics. To examine whether the shared representation captures action-related information, we introduce a lightweight motion probe that shows the frozen visual representation contains recoverable motion information. Thinker-WM is trained on a heterogeneous corpus comprising approximately 20K hours of egocentric video, 10K hours of real-robot data, and 1K hours of simulation data, including approximately 4K hours of real-robot data with aligned depth supervision. Extensive experiments demonstrate strong performance across embodied world-model evaluations. Thinker-WM ranks first on EWMBench, WorldOlympiad and WorldArena, while controlled studies further demonstrate the benefits of action conditioning, geometric interaction, and multi-function training within the shared representation.
Related papers
- EWAM: Emergent Depth-Wise Specialization in a Unified Embodied Model -- From Semantic Understanding through Visual Foresight to Action
- PrismWAM: Decoupling State Representations in World Action Models
- UniWAM: Unified World-Action Model
- Magic-W0: A Structured World-Action Foundation Model for Physical Intelligence
- UniWALM: Unified World Action Language Model for Embodied Robot Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-14 UTC
Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence
Preprint posted: 2026-10-13 · Original record ↗
- arxiv
- 2610.18447
Explicit source versions (1)
- v3 · 2026-10-19
- Preprint available
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · OAI arXiv created; not header datestamp (official metadata)
- Official evidence ↗ · atom:published (official metadata)
- Explicit source version detected · v3
Source date: 2026-10-19 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
Abstract · arxiv
World models have emerged as a promising paradigm for embodied AI, yet existing systems often remain fragmented in the modalities they model and the functions they support. In particular, visual dynamics, geometric structure, and executable action are commonly coupled through separate interfaces or downstream modules rather than being jointly represented and evolved within a shared dynamics model. In this work, we present Thinker-WM, a unified multimodal world model that jointly models and generates RGB, metric depth, and robot action through a shared intermediate Unified Dynamics Transformer with modality-specific encoders and decoders. The model performs multimodal unification at the representation level: appearance, geometry, and action are transformed into a common dynamic representation and jointly evolved by shared parameters, while their distinct output structures are preserved. The same parameterization supports multiple functions, including joint video-action generation, world prediction, policy inference, action-conditioned generation, video-to-action generation, and inverse dynamics. To examine whether the shared representation captures action-related information, we introduce a lightweight motion probe that shows the frozen visual representation contains recoverable motion information. Thinker-WM is trained on a heterogeneous corpus comprising approximately 20K hours of egocentric video, 10K hours of real-robot data, and 1K hours of simulation data, including approximately 4K hours of real-robot data with aligned depth supervision. Extensive experiments demonstrate strong performance across embodied world-model evaluations. Thinker-WM ranks first on EWMBench, WorldOlympiad and WorldArena, while controlled studies further demonstrate the benefits of action conditioning, geometric interaction, and multi-function training within the shared representation.