Video World Models · 2026-10-21
Learning the 4D World through Point Trajectory Forecasting
Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim, Jonghoon Lee, Jungwoo Park, Inhee Lee, Doosan Baek, Jinil Kim, Taehyun Lim, Hanbyul Joo, Marc Pollefeys, Saining Xie, Jinwoo Shin
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce Video-Point Model (VPM), which jointly forecasts future multi-view videos and the 3D trajectories of persistent scene points from arbitrary viewpoints, conditioned on multi-view video histories and a language instruction. They extend a pretrained world model with camera-conditioned multi-view processing and add a trajectory head, training with VPM-Data and evaluating on VP-WorldBench. They further add robot action tokens and show improved manipulation success with point supervision.
Why it matters
The paper targets a core gap in multi-view forecasting: video plausibility alone does not guarantee coherent 3D motion. By supervising trajectories of persistent 3D points alongside video generation, the model is pushed to anchor appearance changes across cameras to shared 3D motion, and the resulting representation can also benefit action-enabled robot control.
Method
- VPM conditions on posed multi-view video histories and a language instruction, and predicts future videos from requested viewpoints plus future 3D trajectories of N persistent points selected from observed frames.
- It integrates cross-camera evidence by packing camera tokens into a shared sequence and uses Plücker-ray camera embeddings and a 4D RoPE view-axis to distinguish camera streams.
- For robot control, it adds robot action tokens as an additional token modality that shares the same point-supervised backbone and is denoised jointly with future videos.
Limitation
The evaluation assumes that no method (including VPM) receives camera parameters for future frames; all methods must infer camera motion from the observed history.
Abstract (from arXiv)
Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time. Each trajectory tracks a single point across all views, and supervising these trajectories alongside video generation therefore anchors appearance changes in every camera to a shared description of 3D motion. To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments. We further extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, thereby grounding the policy in the same 3D-aware representation. On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model.
Related papers
- WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
- PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
- Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models
- LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC
Learning the 4D World through Point Trajectory Forecasting
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28594
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time. Each trajectory tracks a single point across all views, and supervising these trajectories alongside video generation therefore anchors appearance changes in every camera to a shared description of 3D motion. To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments. We further extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, thereby grounding the policy in the same 3D-aware representation. On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model.