Video World Models · 2026-11-03
AuroraWorld: Structured 4D World Modeling for Exo-to-Ego Video Generation in Robot Manipulation
Hao Xue, Peng-Fei Zhang, Ying Cheng, Lanyun Zhu, Lei Zhu, Heng Tao Shen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AuroraWorld is a structured 4D world model for exo-to-ego video generation in robot manipulation. It infers a structured 4D embodied representation from sparse exocentric observations using spatial (3D geometric tokens), temporal (motion-stabilized keyframes), and motion (inferred end-effector trajectories) components, then uses these as priors for diffusion-based egocentric video generation. The authors evaluate on visual generation and downstream embodied policy learning in simulation and real robotic environments.
Why it matters
The paper targets the data bottleneck of egocentric robot videos by synthesizing robot-centric (wrist) views from large-scale exocentric videos. The authors’ key claim is that structured spatial/temporal/motion priors improve physically consistent and temporally coherent egocentric generation under sparse exocentric observations, which in turn improves downstream policy learning.
Method
- Infer a structured 4D embodied representation W^{4D} from sparse exocentric videos: spatial structures W^{sp} from latent 3D scene tokens, temporal keyframes W^{tp} via motion-stabilized anchor selection, and motion dynamics W^{mo} via inferred 8-D end-effector trajectories.
- Adapt LagerNVS to transform selected exocentric frames into egocentric keyframes using predicted target egocentric camera pose from the inferred end-effector state.
- Use the inferred 4D representation as structured priors for a conditional latent diffusion video model, including sparse temporal anchor conditioning and spatial-motion guided latent initialization.
Limitation
AuroraWorld relies on upstream predictions (including IDM trajectory, temporal-anchor selection, target-camera prediction, and structured-prior construction) whose errors may propagate. The paper also states that sparse exocentric observations may be insufficient under severe occlusion, rapid contact dynamics, or tasks requiring fine-grained spatial alignment, and that policies trained with synthetic egocentric observations do not consistently match those trained with recorded egocentric views.
Abstract (from arXiv)
Egocentric observations provide a robot-centric view of manipulation processes, capturing contact interactions and object-state changes that are important for visuomotor policy learning but difficult to observe from external cameras. However, collecting such data at scale remains expensive due to additional hardware requirements and platform-specific deployment costs. Synthesizing egocentric observations from large-scale exocentric manipulation videos provides a scalable alternative. However, existing methods struggle to synthesize physically consistent and policy-relevant egocentric videos under substantial viewpoint transformations, severe occlusions, complex object interactions, and inherently sparse observations from available exocentric viewpoints. Code will be made publicly available at https://github.com/xueha0/AuroraWorld
Related papers
- Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
- World Action Learning via Interaction-Centric Spectral Latent Guidance
- EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
- Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
- Open-HumanWAM: An Open Recipe for Scaling Human and Robot Experience for Embodied Foundation Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC
AuroraWorld: Structured 4D World Modeling for Exo-to-Ego Video Generation in Robot Manipulation
First announced by arXiv: 2026-11-03 · Original record ↗
- arxiv
- 2611.00177
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.00177v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Egocentric observations provide a robot-centric view of manipulation processes, capturing contact interactions and object-state changes that are important for visuomotor policy learning but difficult to observe from external cameras. However, collecting such data at scale remains expensive due to additional hardware requirements and platform-specific deployment costs. Synthesizing egocentric observations from large-scale exocentric manipulation videos provides a scalable alternative. However, existing methods struggle to synthesize physically consistent and policy-relevant egocentric videos under substantial viewpoint transformations, severe occlusions, complex object interactions, and inherently sparse observations from available exocentric viewpoints. Code will be made publicly available at https://github.com/xueha0/AuroraWorld
