Video World Models · 2026-11-06
GPWM: Modeling the World as 3D Particle Motion
Robert Ren, Matthew Bronars, He Zhu, Yikai Tang, Jianing Zhang, Sergey Zakharov, Pavel Tokmakov, Katerina Fragkiadaki
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
GPWM is a generative 3D physical world model that predicts the motion of particles and rigid bodies directly in physical space, using an object-aware multi-resolution transformer with flow matching and diffusion forcing. Trained on ParticleWorlds (25.5M frames), it reduces prediction error by 31% on average over the strongest applicable learned-dynamics baselines across six physical domains and enables contact-rich planning by searching in latent noise space.
Why it matters
The authors address learned 3D dynamics that are often domain-specific or limited by deterministic prediction/short horizons. GPWM unifies rigid, deformable, cloth, and granular dynamics in a single architecture and supports long-horizon streaming generation. It also enables planning by learning a generative prior over actuator–object interactions and optimizing latent noise, improving over action-space search on pushing and picking.
Method
- Represent scenes with particle displacements for deformable motion and rigid-body pose transformations; use object-aware multi-resolution spatio-temporal transformer with factorized attention (temporal then spatial) in relative 3D coordinates.
- Train with flow matching plus diffusion forcing using independent per-future-frame noise levels, and generate with a staggered denoising schedule to stream beyond the training horizon.
- For planning, jointly model actuator and passive-object trajectories; perform latent noise optimization (LNO) or CEM in the generative model’s noise space guided by sparse rewards.
Abstract (from arXiv)
We introduce GPWM, a Generative Particle World Model that predicts how 3D scenes evolve by generating the motion of particles and rigid bodies over time. Unlike video world models that generate scene appearance, GPWM operates directly in 3D physical space. A single representation and architecture cover rigid, deformable, cloth, and granular materials, in both passive physics and robotic interaction. GPWM is an object-aware multi-resolution transformer that combines flow matching with diffusion forcing to generate coherent physical trajectories far beyond its training horizon. We train and evaluate GPWM on ParticleWorlds, a 25.5-million-frame corpus spanning diverse geometries, materials, interactions, and robot embodiments. GPWM outperforms the best prior learned-dynamics baseline in each of six physical domains, reducing prediction error by 31% on average. Breaking from the domain-specific setting of prior methods, a single Unified GPWM models these heterogeneous regimes with one set of weights, largely outperforming per-domain baselines and even surpassing our own specialist models in two domains. GPWM further generalizes to unseen geometries, physical properties, object counts, and rollout horizons. Beyond prediction, GPWM enables contact-rich planning by learning a generative prior over joint actuator-object motion. Instead of searching over unconstrained actions, planning searches the model's latent noise for interactions that achieve a desired object-space goal. With the same world model, noise-space planning substantially outperforms action-space search on pushing and picking, where complex contact sequences are hard to discover. Executed open-loop in simulation, the resulting plans achieve goal-directed manipulation without reference action trajectories or goal-specific policies.
Related papers
- Hybrid Particle-Voxel Dynamics for Multimodal 3D World Simulation
- WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation
- Action Representation for Spatial Generalization in Robot World Models
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-05 · Indexed by Pub-AI: 2026-11-06 UTC
GPWM: Modeling the World as 3D Particle Motion
Preprint posted: 2026-11-05 · Original record ↗
- arxiv
- 2611.08338
Explicit source versions (1)
- v1 · 2026-11-05
- Preprint available
Source date: 2026-11-05 · First detected: 2026-11-06 UTC · Historical baseline
Source evidence (3)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.08338v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
We introduce GPWM, a Generative Particle World Model that predicts how 3D scenes evolve by generating the motion of particles and rigid bodies over time. Unlike video world models that generate scene appearance, GPWM operates directly in 3D physical space. A single representation and architecture cover rigid, deformable, cloth, and granular materials, in both passive physics and robotic interaction. GPWM is an object-aware multi-resolution transformer that combines flow matching with diffusion forcing to generate coherent physical trajectories far beyond its training horizon. We train and evaluate GPWM on ParticleWorlds, a 25.5-million-frame corpus spanning diverse geometries, materials, interactions, and robot embodiments. GPWM outperforms the best prior learned-dynamics baseline in each of six physical domains, reducing prediction error by 31% on average. Breaking from the domain-specific setting of prior methods, a single Unified GPWM models these heterogeneous regimes with one set of weights, largely outperforming per-domain baselines and even surpassing our own specialist models in two domains. GPWM further generalizes to unseen geometries, physical properties, object counts, and rollout horizons. Beyond prediction, GPWM enables contact-rich planning by learning a generative prior over joint actuator-object motion. Instead of searching over unconstrained actions, planning searches the model's latent noise for interactions that achieve a desired object-space goal. With the same world model, noise-space planning substantially outperforms action-space search on pushing and picking, where complex contact sequences are hard to discover. Executed open-loop in simulation, the resulting plans achieve goal-directed manipulation without reference action trajectories or goal-specific policies.
