Video World Models · 2026-10-19

Persistent Planar Memory for Video World Models

Yuze He, Bin Tan, Zelin Gao, Yujun Shen, Yong-Jin Liu, Nan Xue, Yinghao Xu

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

PPMem is a video world model that maintains a persistent 3D memory using plane primitives. It augments a video diffusion transformer with a planar prediction head and a Depth Token Fusion module: each autoregressive chunk jointly generates RGB and plane geometry, merges planes into a persistent pool, renders depth from the pool, and uses it to condition later chunks. The authors report much better cross-frame coherence and video quality on DL3DV and RealEstate10K, plus metric depth, normals, and exportable planar geometry.

Why it matters

The paper targets a long-context “persistent 3D state” gap in camera-controlled video world models. The authors propose an explicit, compact, geometrically explicit memory (oriented planes) that is updated over time and used for cross-chunk conditioning, aiming to preserve coherence under large viewpoint changes.

Method

PPMem uses plane primitives (7 parameters each) as a persistent 3D plane pool; predicted planes are accumulated via a merging procedure across autoregressive chunks.
Architecture: a video diffusion transformer with (1) a plane prediction head adapted from Plana3R and (2) Depth Token Fusion (DTF) that injects rendered plane-pool depth into the transformer features.
At each new chunk, the current plane pool is rendered into partial depth maps used for conditioning; geometry is decoded into per-frame plane primitives that merge into the pool.

Abstract (from arXiv)

Video world models that generate and explore environments at scale need a persistent 3D memory of what they have produced. Plane primitives offer a natural candidate: a compact set of oriented planes captures the dominant structural elements (e.g., walls, floors, facades) of indoor and outdoor scenes, while requiring only 7 parameters per primitive. We present PPMem, a video world model that adopts plane primitives as its persistent 3D memory. PPMem augments a video diffusion transformer with a planar prediction head and a Depth Token Fusion (DTF) module: each autoregressive chunk jointly produces RGB video and planar geometry, which merges into a persistent plane memory that conditions subsequent chunks via rendered depth. The resulting memory is three orders of magnitude more compact than per-pixel point clouds (~100K primitives, 5MB for a full scene), geometrically explicit (metric depth, normals, exportable surfaces), cross-view consistent (jointly predicted by the shared backbone), and adaptive (refined by the merging procedure as generation proceeds). On DL3DV and RealEstate10K, PPMem outperforms prior 3D-grounded world models in video quality while jointly producing metric-scale depth, surface normals, and exportable planar geometry, sustaining coherence across hundreds of frames under large viewpoint changes. Project page: https://hyzcluster.github.io/ppmem.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-16 · Indexed by Pub-AI: 2026-10-19 UTC

preprint