Video World Models · 2026-10-30
Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling
Haitang Feng, Zhuoman Liu, Zhennan Wang, Sheng Wang, Juelin Li, Chengzhi Zhao, Hexiang Wei, Lin Shao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
GenSight is a unified embodied world model that combines future video generation and dense 4D perception (depth, embodiment segmentation, and first-frame-relative 2D flow) in a single pretrained video diffusion transformer. It uses a shared DiT backbone with lightweight task modulation, a common spatiotemporal (image-like) representation, and joint training in a unified velocity prediction space. In generation it produces future video with aligned dense predictions; in perception it predicts dense modalities in one step from an observed video.
Why it matters
The authors target embodied world modeling where the model should both imagine plausible futures and perceive structured geometry, semantics, and motion. GenSight avoids separate generation-then-estimation pipelines by unifying modality representation, architecture, and prediction space, aiming to couple appearance, geometry, semantics, and motion and improve cross-modal consistency through shared spatiotemporal priors.
Method
- Unified model: a single pretrained Video DiT shared across tasks, with task-adaptive modulation via Adaptive RMSNorm (shared Transformer weights; lightweight per-task modulation).
- Everything-as-RGB representation: video, depth, embodiment segmentation, and first-frame-relative 2D flow are mapped to three-channel RGB-like sequences using deterministic encoding/decoding interfaces; the same frozen video autoencoder is used for all modalities.
- Unified velocity prediction space: generation and dense perception are jointly trained to predict velocities for RGB, depth, segmentation, and flow, supervised by velocity regression (plus spatial-gradient matching for dense modalities) and enabled by two inference modes (multi-step generation with dense readouts; single-step dense perception).
Limitation
The text does not state a limitation specific to the method's performance, robustness, compute, or failure cases (null).
Abstract (from arXiv)
Embodied world models should not only imagine plausible futures, but also perceive the geometric, semantic, and motion structure of the evolving world. We present GenSight, a unified embodied world model that brings future video generation and dense perception into a single pretrained video diffusion transformer. GenSight uses a unified spatiotemporal representation for video, depth, embodiment segmentation, and first-frame-relative 2D flow. Built on a shared DiT backbone, GenSight uses lightweight task modulation to predict all modalities within the same architecture. All tasks are jointly optimized within a unified velocity prediction space. At inference, GenSight supports two complementary modes: multi-step future video generation with aligned dense predictions and single-step dense perception from an observed video. By unifying modality representation, model architecture, and prediction space, GenSight avoids separate modality-specific pipelines and couples appearance, geometry, semantics, and motion in a shared representation. It promotes cross-modal consistency and allows dense perception to benefit from the spatiotemporal priors learned through video generation. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both video generation and dense 4D perception, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.
Related papers
- Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence
- ReactWorld: Structuring Interaction Responses for Embodied Video Generation
- WorldFoundry: Towards Unified Infrastructure for World Intelligence
- EWAM: Emergent Depth-Wise Specialization in a Unified Embodied Model -- From Semantic Understanding through Visual Foresight to Action
- WorldCast: Exploring Intelligent Visual Generation in World State Prediction
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC
Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling
Preprint posted: 2026-10-29 · Original record ↗
- arxiv
- 2610.39232
Explicit source versions (1)
- v1 · 2026-10-29
- Preprint available
Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.39232v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Embodied world models should not only imagine plausible futures, but also perceive the geometric, semantic, and motion structure of the evolving world. We present GenSight, a unified embodied world model that brings future video generation and dense perception into a single pretrained video diffusion transformer. GenSight uses a unified spatiotemporal representation for video, depth, embodiment segmentation, and first-frame-relative 2D flow. Built on a shared DiT backbone, GenSight uses lightweight task modulation to predict all modalities within the same architecture. All tasks are jointly optimized within a unified velocity prediction space. At inference, GenSight supports two complementary modes: multi-step future video generation with aligned dense predictions and single-step dense perception from an observed video. By unifying modality representation, model architecture, and prediction space, GenSight avoids separate modality-specific pipelines and couples appearance, geometry, semantics, and motion in a shared representation. It promotes cross-modal consistency and allows dense perception to benefit from the spatiotemporal priors learned through video generation. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both video generation and dense 4D perception, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.
