Video World Models · 2026-10-07
PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation
Yu Zhang, Xudong Xu, Xingang Pan
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
PhysLDM is a latent-diffusion paradigm for one-shot prediction of high-fidelity volumetric deformable-body trajectories. It uses a holistic spatiotemporal VAE (ST-VAE) that avoids “staircase” artifacts and decouples rigid motion from local deformation, achieving ~2.48 mm reconstruction precision at up to 78× token compression. In chaotic dynamics, diffusion models physically plausible multi-modal futures better than deterministic regression. A single kinematics-only model generalizes zero-shot to unseen OOD datasets and supports differentiable inverse problems.
Why it matters
Long-horizon, high-resolution deformable simulation is hard for autoregressive methods (error accumulation) and direct multi-frame prediction (compute cost). PhysLDM addresses both via a compact spatiotemporal latent representation and a diffusion-based one-shot predictor, and argues diffusion is better suited than regression in chaotic regimes. It also claims constitutive-model-free supervision, zero-shot generalization, and differentiability for inverse problems and higher-order design optimization.
Method
- Holistic spatiotemporal VAE (ST-VAE): encodes per-frame canonical deformations with a Spatial VAE and compresses the T-frame trajectory using factorized self-attention (holistic temporal tokens) to avoid “staircase” artifacts and decouple rigid motion from deformation.
- Latent diffusion one-shot prediction: an LDM conditioned on the initial state predicts the entire latent trajectory in a single diffusion pass; geometry is decoded by a frozen ST-VAE decoder and refined via a GNN over the tetrahedral mesh.
- Kinematics-only training and differentiation: reconstruction losses are computed from vertex positions/velocities/rigid poses (no constitutive models), and the differentiable model supports higher-order derivatives for inverse problems and design optimization.
Limitation
The authors report that diffusion performs better than deterministic regression in chaotic dynamics, and discuss that zero-shot performance depends on domain; for real captured point clouds, the LDM prediction fails in zero-shot (875 mm) because of a domain gap (object manipulation on a table vs objects thrown onto the ground).
Abstract (from arXiv)
Neural simulation of high-fidelity deformable bodies is a foundational challenge in computer graphics and physical AI. Long-horizon prediction for high-resolution 3D volumetric meshes is hard: autoregressive methods are susceptible to error accumulation, while direct multi-frame prediction at native resolution is computationally prohibitive. This motivates a compact spatiotemporal latent representation, which is largely unexplored for mesh-based volumetric physics. Meanwhile, it remains unclear whether deterministic regression or generative diffusion is the more appropriate predictive paradigm. To address these coupled challenges, we introduce PhysLDM, a unified latent-diffusion paradigm for one-shot volumetric deformable simulation. Its core is a holistic spatiotemporal VAE that avoids the "staircase" artifacts of standard temporal compression (as in common video VAEs), achieving ~2.48 mm reconstruction precision on meter-scale scenes at up to 78x token compression. Based on this reliable latent space, we systematically compare regression and diffusion methods. Our experiments uncover a key modeling insight: complex deformable dynamics are often chaotic, and in this regime deterministic regression tends to produce non-physical averages, whereas diffusion better models their distribution. Accordingly, we employ a latent diffusion model that effectively learns from the chaotic data to generate physically plausible trajectories. Trained purely kinematically on an Objaverse-scale dataset, a single PhysLDM generalizes zero-shot to unseen OOD datasets (GSO and Toys4K). Its differentiability further enables efficient solution of inverse problems and higher-order design optimization. To our knowledge, PhysLDM is the first high-fidelity spatiotemporal autoencoder and latent-diffusion paradigm for volumetric deformable dynamics, offering a scalable and robust approach to neural simulation.
Related papers
- Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models
- Glance Forcing: Making Bidirectional Diffusion Autoregressive at a Glance
- Structured Latent Dynamics for Long-Horizon Embodied World Models
- Identifiable World Models from Pretrained Diffusion Representations
- Decoupled Drift-Diffusion World Models for Consistent Future Frame Prediction
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC
PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation
Preprint posted: 2026-10-06 · Original record ↗
- arxiv
- 2610.07609
Explicit source versions (1)
- v3 · 2026-10-18
- Explicit source version detected · v3
Source date: 2026-10-18 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Preprint available
Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Neural simulation of high-fidelity deformable bodies is a foundational challenge in computer graphics and physical AI. Long-horizon prediction for high-resolution 3D volumetric meshes is hard: autoregressive methods are prone to error accumulation, while direct multi-frame prediction at native resolution is computationally prohibitive. This motivates a compact spatiotemporal latent representation, which is largely unexplored for mesh-based volumetric physics. Meanwhile, it remains unclear whether deterministic regression or generative diffusion is the more suitable predictive paradigm. To address these coupled challenges, we introduce PhysLDM, a unified latent-diffusion paradigm for one-shot deformable simulation. Its core is a holistic spatiotemporal VAE that avoids the "staircase" artifacts of standard temporal compression (e.g., video VAEs), achieving ~2.48 mm reconstruction precision on meter-scale scenes at up to 78x token compression. Based on this reliable latent space, we systematically compare regression and diffusion methods. Our experiments uncover a key modeling insight: complex deformable dynamics are often chaotic, and in this regime deterministic regression tends to produce non-physical averages, whereas diffusion better models their distribution. Accordingly, we employ a latent diffusion model that effectively learns from chaotic data to generate physically plausible trajectories. Trained purely kinematically on an Objaverse-scale dataset, a single PhysLDM generalizes zero-shot to unseen OOD datasets. Its differentiability further enables efficient solution of inverse problems and higher-order design optimization. To our knowledge, PhysLDM is the first high-fidelity spatiotemporal autoencoder and latent-diffusion paradigm for volumetric deformable dynamics, offering a scalable and robust approach to neural simulation. Project Page: https://yucrazing.github.io/physldm/