Video World Models · 2026-10-02
4Director: Controlling Video World Models with Rigid 3D Geometry
Wei Cao, Hao Zhang, Vikram Voleti, Yuqun Wu, Mallikarjun B R, Shimon Vainer, Mark Boss, Yaoyao Liu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
4Director is a video world model for controlling camera and objects using an explicit 4D scene representation. Each object is reconstructed once as a complete canonical mesh and moved by one prescribed rigid transformation per frame in a shared coordinate frame with the camera. A Motion Adapter injects a depth-rendered rigid scaffold into a pretrained video generator to synthesize view-consistent appearance, illumination, and non-rigid dynamics. The paper introduces RealCOD-Rigid (20,774 clips) and Identity-Gated IoU (IG-IoU).
Why it matters
The authors target ambiguity and incomplete geometry in prior controls, by exposing an editable 4D scene with complete object meshes and per-frame rigid transformations for both camera and objects. They evaluate joint camera/object control with visual quality metrics and a control metric (IG-IoU) that credits mask overlap only when identity is preserved.
Method
- Lift the input image to a shared 3D coordinate frame using MoGe-2 (depth/intrinsics) and SAM2 (object masks); reconstruct each marked object once as a complete canonical mesh (Pixal3D) and lift background into a static point cloud; user edits camera and per-object rigid trajectories.
- For each frame, render a depth video control by projecting the camera and the rigidly transformed meshes/background into a depth map (nearest surface per pixel).
- Train a Motion Adapter (DiT-style branch) on top of a pretrained video generator (Wan2.1-VACE-14B): condition on the depth-rendered control, input image, and prompt, and have the generator supply appearance, illumination, and non-rigid dynamics.
Limitation
One rigid transformation per object cannot express motion that is mostly articulated; the authors show a breakdancer moved as one rigid body keeps the handstand pose instead of dancing.
Abstract (from arXiv)
Precise control over camera and object motion is essential for professional video production. Existing methods control objects only coarsely, through image-plane cues that are ambiguous in depth and rotation or through 3D tracks and blobs that lack complete geometry and lose consistency across viewpoint changes. We introduce 4Director, a video world model conditioned on an explicit 4D scene representation: each object is reconstructed once from the input image as a canonical mesh and moved by one prescribed rigid transformation per frame. This representation provides an intuitive 3D control interface and prevents unobserved geometry from being regenerated independently in every frame. We render the controlled scene as a depth video and introduce a Motion Adapter that transforms this geometric scaffold into video while synthesizing view-consistent appearance, illumination, and non-rigid dynamics. For training, we construct RealCOD-Rigid, a new dataset of 20,774 clips annotated with rigid 3D scenes by our automatic pipeline. We further introduce Identity-Gated IoU (IG-IoU), which jointly evaluates adherence to prescribed object motion and preservation of object identity. Experiments demonstrate that 4Director consistently outperforms prior methods in visual quality and in camera and object control.
Related papers
- Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution
- Demiurge: Forging Navigable 4D Worlds via Autoregressive Video Rerendering
- OuroWorld: Bringing Any 3D World Alive as Diverse, Endlessly Looping 3D Cinemagraphs
- Learning the 4D World through Point Trajectory Forecasting
- DepthWorld: 3D World Model for Robot Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC
4Director: Controlling Video World Models with Rigid 3D Geometry
Preprint posted: 2026-10-01 · Original record ↗
- arxiv
- 2610.02160
Explicit source versions (1)
- v1 · 2026-10-01
- Preprint available
Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Precise control over camera and object motion is essential for professional video production. Existing methods control objects only coarsely, through image-plane cues that are ambiguous in depth and rotation or through 3D tracks and blobs that lack complete geometry and lose consistency across viewpoint changes. We introduce 4Director, a video world model conditioned on an explicit 4D scene representation: each object is reconstructed once from the input image as a canonical mesh and moved by one prescribed rigid transformation per frame. This representation provides an intuitive 3D control interface and prevents unobserved geometry from being regenerated independently in every frame. We render the controlled scene as a depth video and introduce a Motion Adapter that transforms this geometric scaffold into video while synthesizing view-consistent appearance, illumination, and non-rigid dynamics. For training, we construct RealCOD-Rigid, a new dataset of 20,774 clips annotated with rigid 3D scenes by our automatic pipeline. We further introduce Identity-Gated IoU (IG-IoU), which jointly evaluates adherence to prescribed object motion and preservation of object identity. Experiments demonstrate that 4Director consistently outperforms prior methods in visual quality and in camera and object control.