Latent Dynamics · 2026-11-09

MotionCore: A Latent Neural Simulator for Video Generation with Rigid-Body Dynamics

Zheng Wang, Zhengwei Gong, Yuanzhi Liang, Haibin Huang, Chi Zhang

arXiv:2611.08904PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

MotionCore learns rigid-body physical dynamics in latent space using a neural operator/PDE-inspired solver, then feeds predicted object trajectories into a controllable video diffusion model via spatially aligned latent controls. The framework uses a diffusion-matching objective plus a low-capacity decodability constraint to preserve trajectory through the visual interface. It evaluates on physical-consistency benchmarks and paired data with known trajectories.

Why it matters

The authors address a gap where realistic video generation can produce physically inconsistent motion. MotionCore separates physical trajectory prediction (learned dynamics) from visual rendering (fixed diffusion generator), aiming for improved physical consistency and trajectory obedience while keeping visual quality comparable. The paper also includes an ablation protocol comparing learned dynamics to an analytic prior and studying latent evolution processors.

Method

  • Learn a latent neural simulator (encoder→latent operator→decoder) to predict future rigid-body trajectories from an observed motion history, approximating rigid dynamics via a PDE formulation using compliant elastic contact laws.
  • Convert predicted physical states into spatially aligned latent controls for a controllable video diffusion generator (VACE built on Wan2.1-1.3B), keeping the generator fixed as the visual interface.
  • Preserve trajectory through the interface using a diffusion-matching objective together with a low-capacity decodability constraint on object identity and position; include analytic-prior and processor-choice ablations.
Abstract (from arXiv)

Video generation models can produce highly realistic videos while generating physically inconsistent object motion. This limitation suggests that visual modeling alone may not provide a transferable representation of physical dynamics. We introduce MotionCore, a framework for learning physical dynamics via a learnable partial differential equation solver and focuses on rigid-body motion. We introduce a latent neural simulator learning to predict the evolution of object-level physical states from observed motion. Its predictions are then converted into spatially aligned latent controls for a controllable video diffusion model. The generator serves as a visual interface: it provides appearance and scene detail, while the learned dynamics model specifies the physical trajectory. To preserve this trajectory through the interface, MotionCore combines a diffusion-matching objective with a low-capacity decodability constraint on object identity and position. We evaluate MotionCore on public physical-consistency benchmarks and on paired data with known trajectories. Results indicate that MotionCore improves physical consistency and trajectory obedience in the evaluated rigid-body settings, while maintaining comparable visual quality. A controlled ablation protocol compares learned dynamics with an analytic prior and studies the choice of latent evolution processor.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-06 · Indexed by Pub-AI: 2026-11-09 UTC

preprint

MotionCore: A Latent Neural Simulator for Video Generation with Rigid-Body Dynamics

Zheng Wang, Zhengwei Gong, Yuanzhi Liang, Chi Zhang

Preprint posted: 2026-11-06 · Original record ↗

arxiv
2611.08904
Explicit source versions (1)
  • v1 · 2026-11-06
  1. Preprint available

    Source date: 2026-11-06 · First detected: 2026-11-09 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Video generation models can produce highly realistic videos while generating physically inconsistent object motion. This limitation suggests that visual modeling alone may not provide a transferable representation of physical dynamics. We introduce MotionCore, a framework for learning physical dynamics via a learnable partial differential equation solver and focuses on rigid-body motion. We introduce a latent neural simulator learning to predict the evolution of object-level physical states from observed motion. Its predictions are then converted into spatially aligned latent controls for a controllable video diffusion model. The generator serves as a visual interface: it provides appearance and scene detail, while the learned dynamics model specifies the physical trajectory. To preserve this trajectory through the interface, MotionCore combines a diffusion-matching objective with a low-capacity decodability constraint on object identity and position. We evaluate MotionCore on public physical-consistency benchmarks and on paired data with known trajectories. Results indicate that MotionCore improves physical consistency and trajectory obedience in the evaluated rigid-body settings, while maintaining comparable visual quality. A controlled ablation protocol compares learned dynamics with an analytic prior and studies the choice of latent evolution processor.