Video World Models · 2026-11-03

AuroraWorld: Structured 4D World Modeling for Exo-to-Ego Video Generation in Robot Manipulation

Hao Xue, Peng-Fei Zhang, Ying Cheng, Lanyun Zhu, Lei Zhu, Heng Tao Shen

arXiv:2611.00177PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AuroraWorld is a structured 4D world model for exo-to-ego video generation in robot manipulation. It infers a structured 4D embodied representation from sparse exocentric observations using spatial (3D geometric tokens), temporal (motion-stabilized keyframes), and motion (inferred end-effector trajectories) components, then uses these as priors for diffusion-based egocentric video generation. The authors evaluate on visual generation and downstream embodied policy learning in simulation and real robotic environments.

Why it matters

The paper targets the data bottleneck of egocentric robot videos by synthesizing robot-centric (wrist) views from large-scale exocentric videos. The authors’ key claim is that structured spatial/temporal/motion priors improve physically consistent and temporally coherent egocentric generation under sparse exocentric observations, which in turn improves downstream policy learning.

Method

  • Infer a structured 4D embodied representation W^{4D} from sparse exocentric videos: spatial structures W^{sp} from latent 3D scene tokens, temporal keyframes W^{tp} via motion-stabilized anchor selection, and motion dynamics W^{mo} via inferred 8-D end-effector trajectories.
  • Adapt LagerNVS to transform selected exocentric frames into egocentric keyframes using predicted target egocentric camera pose from the inferred end-effector state.
  • Use the inferred 4D representation as structured priors for a conditional latent diffusion video model, including sparse temporal anchor conditioning and spatial-motion guided latent initialization.

Limitation

AuroraWorld relies on upstream predictions (including IDM trajectory, temporal-anchor selection, target-camera prediction, and structured-prior construction) whose errors may propagate. The paper also states that sparse exocentric observations may be insufficient under severe occlusion, rapid contact dynamics, or tasks requiring fine-grained spatial alignment, and that policies trained with synthetic egocentric observations do not consistently match those trained with recorded egocentric views.

Abstract (from arXiv)

Egocentric observations provide a robot-centric view of manipulation processes, capturing contact interactions and object-state changes that are important for visuomotor policy learning but difficult to observe from external cameras. However, collecting such data at scale remains expensive due to additional hardware requirements and platform-specific deployment costs. Synthesizing egocentric observations from large-scale exocentric manipulation videos provides a scalable alternative. However, existing methods struggle to synthesize physically consistent and policy-relevant egocentric videos under substantial viewpoint transformations, severe occlusions, complex object interactions, and inherently sparse observations from available exocentric viewpoints. Code will be made publicly available at https://github.com/xueha0/AuroraWorld

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

AuroraWorld: Structured 4D World Modeling for Exo-to-Ego Video Generation in Robot Manipulation

Hao Xue, Peng-Fei Zhang, Ying Cheng, Lanyun Zhu, Lei Zhu, Heng Tao Shen

First announced by arXiv: 2026-11-03 · Original record ↗

arxiv
2611.00177
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.00177v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Egocentric observations provide a robot-centric view of manipulation processes, capturing contact interactions and object-state changes that are important for visuomotor policy learning but difficult to observe from external cameras. However, collecting such data at scale remains expensive due to additional hardware requirements and platform-specific deployment costs. Synthesizing egocentric observations from large-scale exocentric manipulation videos provides a scalable alternative. However, existing methods struggle to synthesize physically consistent and policy-relevant egocentric videos under substantial viewpoint transformations, severe occlusions, complex object interactions, and inherently sparse observations from available exocentric viewpoints. Code will be made publicly available at https://github.com/xueha0/AuroraWorld