Video World Models · 2026-10-30

WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

Yuhao Lu, Jiongxu Chen, Hexiang Wei, Jiajin Wu, Juelin Li, Mingjian Liang, Fan Yang, Zhuowei Liang, Zihao Liu, Zhennan Wang, Sheng Wang, Lin Shao

arXiv:2610.39278 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

WorldFlow is a generative world model that jointly predicts future video and query-based 3D trajectories. It represents 3D motion as continuous displacement tokens aligned to the video grid and uses a unified Mixture-of-Transformers backbone with a rectified-flow objective to denoise both modalities. Trained on ~1.44M object-level samples, it achieves the lowest ADE/FDE among evaluated learning-based methods on WorldFlowBench, including SpatialCog (ADE 0.060 m).

Why it matters

The authors target limitations of prior video world models that leave 3D dynamics implicit, action-conditioned models tied to robot configurations, and flow-only 3D motion prediction with limited physical-dynamics learning signal. By coupling explicit geometric 3D motion flow with dense visual supervision in a single generative model, WorldFlow aims to improve 3D motion forecasting using pretraining-style video modeling signals.

Method

A unified generative world model that jointly predicts future RGB video and query-conditioned 3D flow/trajectories from an observed image, language instruction, and 3D query points.
Parameterizes 3D motion as continuous displacement tokens aligned to the video grid and uses a Mixture-of-Transformers backbone with a rectified-flow objective to denoise both modalities.
Trains with a weighted joint flow matching objective plus auxiliary motion losses (direction, cumulative displacement consistency, and relative endpoint error) and evaluates on WorldFlowBench.

Abstract (from arXiv)

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2\% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9\% and 68.5\%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC

preprint