Video World Models · 2026-11-06

GPWM: Modeling the World as 3D Particle Motion

Robert Ren, Matthew Bronars, He Zhu, Yikai Tang, Jianing Zhang, Sergey Zakharov, Pavel Tokmakov, Katerina Fragkiadaki

arXiv:2611.08338PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

GPWM is a generative 3D physical world model that predicts the motion of particles and rigid bodies directly in physical space, using an object-aware multi-resolution transformer with flow matching and diffusion forcing. Trained on ParticleWorlds (25.5M frames), it reduces prediction error by 31% on average over the strongest applicable learned-dynamics baselines across six physical domains and enables contact-rich planning by searching in latent noise space.

Why it matters

The authors address learned 3D dynamics that are often domain-specific or limited by deterministic prediction/short horizons. GPWM unifies rigid, deformable, cloth, and granular dynamics in a single architecture and supports long-horizon streaming generation. It also enables planning by learning a generative prior over actuator–object interactions and optimizing latent noise, improving over action-space search on pushing and picking.

Method

  • Represent scenes with particle displacements for deformable motion and rigid-body pose transformations; use object-aware multi-resolution spatio-temporal transformer with factorized attention (temporal then spatial) in relative 3D coordinates.
  • Train with flow matching plus diffusion forcing using independent per-future-frame noise levels, and generate with a staggered denoising schedule to stream beyond the training horizon.
  • For planning, jointly model actuator and passive-object trajectories; perform latent noise optimization (LNO) or CEM in the generative model’s noise space guided by sparse rewards.
Abstract (from arXiv)

We introduce GPWM, a Generative Particle World Model that predicts how 3D scenes evolve by generating the motion of particles and rigid bodies over time. Unlike video world models that generate scene appearance, GPWM operates directly in 3D physical space. A single representation and architecture cover rigid, deformable, cloth, and granular materials, in both passive physics and robotic interaction. GPWM is an object-aware multi-resolution transformer that combines flow matching with diffusion forcing to generate coherent physical trajectories far beyond its training horizon. We train and evaluate GPWM on ParticleWorlds, a 25.5-million-frame corpus spanning diverse geometries, materials, interactions, and robot embodiments. GPWM outperforms the best prior learned-dynamics baseline in each of six physical domains, reducing prediction error by 31% on average. Breaking from the domain-specific setting of prior methods, a single Unified GPWM models these heterogeneous regimes with one set of weights, largely outperforming per-domain baselines and even surpassing our own specialist models in two domains. GPWM further generalizes to unseen geometries, physical properties, object counts, and rollout horizons. Beyond prediction, GPWM enables contact-rich planning by learning a generative prior over joint actuator-object motion. Instead of searching over unconstrained actions, planning searches the model's latent noise for interactions that achieve a desired object-space goal. With the same world model, noise-space planning substantially outperforms action-space search on pushing and picking, where complex contact sequences are hard to discover. Executed open-loop in simulation, the resulting plans achieve goal-directed manipulation without reference action trajectories or goal-specific policies.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-05 · Indexed by Pub-AI: 2026-11-06 UTC

preprint

GPWM: Modeling the World as 3D Particle Motion

Robert Ren, Matthew Bronars, He Zhu, Yikai Tang, Jianing Zhang, Sergey Zakharov, Pavel Tokmakov, Katerina Fragkiadaki

Preprint posted: 2026-11-05 · Original record ↗

arxiv
2611.08338
Explicit source versions (1)
  • v1 · 2026-11-05
  1. Preprint available

    Source date: 2026-11-05 · First detected: 2026-11-06 UTC · Historical baseline

    Source evidence (3)
Abstract · arxiv

We introduce GPWM, a Generative Particle World Model that predicts how 3D scenes evolve by generating the motion of particles and rigid bodies over time. Unlike video world models that generate scene appearance, GPWM operates directly in 3D physical space. A single representation and architecture cover rigid, deformable, cloth, and granular materials, in both passive physics and robotic interaction. GPWM is an object-aware multi-resolution transformer that combines flow matching with diffusion forcing to generate coherent physical trajectories far beyond its training horizon. We train and evaluate GPWM on ParticleWorlds, a 25.5-million-frame corpus spanning diverse geometries, materials, interactions, and robot embodiments. GPWM outperforms the best prior learned-dynamics baseline in each of six physical domains, reducing prediction error by 31% on average. Breaking from the domain-specific setting of prior methods, a single Unified GPWM models these heterogeneous regimes with one set of weights, largely outperforming per-domain baselines and even surpassing our own specialist models in two domains. GPWM further generalizes to unseen geometries, physical properties, object counts, and rollout horizons. Beyond prediction, GPWM enables contact-rich planning by learning a generative prior over joint actuator-object motion. Instead of searching over unconstrained actions, planning searches the model's latent noise for interactions that achieve a desired object-space goal. With the same world model, noise-space planning substantially outperforms action-space search on pushing and picking, where complex contact sequences are hard to discover. Executed open-loop in simulation, the resulting plans achieve goal-directed manipulation without reference action trajectories or goal-specific policies.