Video World Models · 2026-11-10
DynOLAT: OLAT Video Prediction for Controllable Dynamic Full-Body Relighting
Taoming Liu, Yinji ShenTu, Guangzhao He, Yudong Jin, Yukang Ye, Siyu Zhang, Fan Jia, Wei Hua, Hujun Bao, Xiaowei Zhou
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DynOLAT is an OLAT video prediction framework for controllable dynamic full-body relighting. It predicts temporally consistent One-Light-At-a-Time (OLAT) basis videos from flat-lit inputs, then linearly combines predicted bases using weights from a target environment map. The authors introduce a dynamic capture pipeline and a large-scale dynamic full-body OLAT dataset (over 24M frames) and report improved lighting-response fidelity and temporal consistency versus tested baselines.
Why it matters
The authors target controllability: lighting direction alignment and relative intensity control in dynamic full-body relighting. By decomposing relighting into directional OLAT basis prediction plus environment-map linear assembly, DynOLAT aims to make directional response explicit and assign relative lighting strength via environment-dependent weights, addressing drifting shadows/highlights and unreliable output brightness reported for prior learning-based relighting methods.
Method
- Predict temporally consistent OLAT basis videos from flat-lit input video latents, then linearly combine predicted bases in color space using target environment-map weights.
- Use Latent Bridge Matching (LBM) to transport flat-lit video latents directly toward OLAT latents, replacing input noise with latent tokens encoded from flat-lit frames to preserve fine textures.
- Condition generation on target lighting direction via block-specific AdaLN modulation and use recurrent bridge context for long-sequence temporal coherence.
Limitation
A key component is a pretrained video VAE; output fidelity is bounded by its reconstruction quality, with noticeable loss of high-frequency details during rapid or large-scale motions. Additionally, the flat-lit input assumption limits direct use on arbitrary consumer footage despite a preliminary in-the-wild delighting module.
Abstract (from arXiv)
Accurate lighting response is essential for realistic dynamic human relighting, because shadows, highlights, and color changes should follow the target illumination in both direction and relative strength. Current learning-based relighting methods can generate visually plausible videos, but they often exhibit imprecise lighting responses: shadows and highlights may drift away from the target direction, and the output brightness may not faithfully follow the relative intensities in the target environment, making the result closer to appearance harmonization than controllable relighting. To address this issue, we propose DynOLAT, a generalizable OLAT video prediction framework for dynamic full-body humans. Instead of directly synthesizing a single relit appearance, our method predicts temporally consistent One-Light-At-a-Time (OLAT) basis videos from flat-lit inputs, and then linearly combines the predicted bases according to the target environment map. This OLAT-based formulation makes the directional response explicit in the predicted bases, while the relative lighting strength is determined by the linear weights derived from the environment map. We further introduce a dynamic capture pipeline and the first large-scale dynamic full-body OLAT dataset, comprising over 24 million frames from 224 identity-outfit combinations, where different outfits of the same person define separate combinations. Experiments show that DynOLAT responds more faithfully to variations in target lighting direction and relative intensity than the tested baselines, enabling controllable video relighting while preserving fine appearance details and temporal coherence. The dataset and processing tools will be publicly released to foster future research.
Related papers
- DreamX: Video Diffusion Model is a Universal Latent Action Learner
- 4Director: Controlling Video World Models with Rigid 3D Geometry
- Cycle WAM: Coupling Action and Video through Cycle Consistency
- Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation
- Glance Forcing: Making Bidirectional Diffusion Autoregressive at a Glance
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
DynOLAT: OLAT Video Prediction for Controllable Dynamic Full-Body Relighting
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.11210
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11210v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Accurate lighting response is essential for realistic dynamic human relighting, because shadows, highlights, and color changes should follow the target illumination in both direction and relative strength. Current learning-based relighting methods can generate visually plausible videos, but they often exhibit imprecise lighting responses: shadows and highlights may drift away from the target direction, and the output brightness may not faithfully follow the relative intensities in the target environment, making the result closer to appearance harmonization than controllable relighting. To address this issue, we propose DynOLAT, a generalizable OLAT video prediction framework for dynamic full-body humans. Instead of directly synthesizing a single relit appearance, our method predicts temporally consistent One-Light-At-a-Time (OLAT) basis videos from flat-lit inputs, and then linearly combines the predicted bases according to the target environment map. This OLAT-based formulation makes the directional response explicit in the predicted bases, while the relative lighting strength is determined by the linear weights derived from the environment map. We further introduce a dynamic capture pipeline and the first large-scale dynamic full-body OLAT dataset, comprising over 24 million frames from 224 identity-outfit combinations, where different outfits of the same person define separate combinations. Experiments show that DynOLAT responds more faithfully to variations in target lighting direction and relative intensity than the tested baselines, enabling controllable video relighting while preserving fine appearance details and temporal coherence. The dataset and processing tools will be publicly released to foster future research.
