Driving · 2026-10-22
HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics
Mauricio Soroco, Yuqiu Liu, Zaid Tasneem, Francesco Pittaluga, Abhishek Aich, Wuyang Chen, Manmohan Chandraker, Ziyu Jiang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
HorizonComposer is a driving video editor that distills and protects traffic semantics during instruction-guided global appearance edits. It builds hybrid paired data from temporally aligned real videos and synthetic edits, trains with offline RL using dense pixel-level rewards, and mitigates “temporal shock” using “thinking frames” for smoother transitions from static conditioning to video dynamics.
Why it matters
Driving scenes require preserving dense traffic semantics (e.g., multi-actor dynamics, lane geometry) during edits. The authors address how general-purpose instruction-guided video editors can corrupt semantics and create spatiotemporal inconsistencies, aiming for structurally reliable edits and improved downstream driving perception performance.
Method
- Data: build a hybrid dataset by pairing temporally aligned real-world driving videos with synthetic edits (using real pairs for structural fidelity and synthetic pairs for unambiguous appearance signals).
- Training: fine-tune via offline RL (CFGRL) with advantage conditioning and extend the indicator to a dense, pixel-aligned mask derived from per-pixel reward maps.
- Model: mitigate temporal shock by prepending “thinking frames” (with temporally reflected driving controls) to warm up generation and reconcile discrepancies before cropping to the original length.
Abstract (from arXiv)
Instruction-guided video generative models offer a scalable solution for stress-testing autonomous driving systems by simulating diverse environmental conditions. However, driving scenes operate within a dense Operational Design Domain (ODD) governed by strict traffic semantics. When applied to global driving-condition edits, general-purpose instruction-guided video editors frequently corrupt these semantics, leading to structural hallucinations and ``temporal shock.'' This work distills and protects traffic semantics during video editing at three levels. First, we construct a hybrid dataset by combining temporally aligned real-world videos with synthetic edits, balancing foundational scene structure with unambiguous appearance signals. Second, during training, we adapt advantage conditioning from offline reinforcement learning to align the model using dense rewards. We extract dense, pixel-level rewards from the hybrid data to serve as a localized signal and actively steer generation toward structurally reliable pixels. Finally, our model mitigates temporal shock by introducing thinking frames, a transition mechanism that seamlessly reconciles static image conditioning with dynamic video context. Extensive experiments demonstrate that our method prevents the corruption of multi-actor dynamics, establishing state-of-the-art spatiotemporal consistency, yielding consistent user preference (39% relative improvement in content preservation and 12% in temporal consistency), and achieving a significant 5% gain in downstream road line segmentation and 7% gain in downstream multi-object detection.
Related papers
- GeoV2V: Geometry-Grounded Video Diffusion Model for Driving Scene Generation
- MORE: Learning Motion-Oriented Registers for Driving World Models and Planning
- DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
- Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Driving
- Cartender: Verifiable Language-Conditioned Driving by Learning the Drives Not Taken
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-22 UTC
HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28829
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Instruction-guided video generative models offer a scalable solution for stress-testing autonomous driving systems by simulating diverse environmental conditions. However, driving scenes operate within a dense Operational Design Domain (ODD) governed by strict traffic semantics. When applied to global driving-condition edits, general-purpose instruction-guided video editors frequently corrupt these semantics, leading to structural hallucinations and ``temporal shock.'' This work distills and protects traffic semantics during video editing at three levels. First, we construct a hybrid dataset by combining temporally aligned real-world videos with synthetic edits, balancing foundational scene structure with unambiguous appearance signals. Second, during training, we adapt advantage conditioning from offline reinforcement learning to align the model using dense rewards. We extract dense, pixel-level rewards from the hybrid data to serve as a localized signal and actively steer generation toward structurally reliable pixels. Finally, our model mitigates temporal shock by introducing thinking frames, a transition mechanism that seamlessly reconciles static image conditioning with dynamic video context. Extensive experiments demonstrate that our method prevents the corruption of multi-actor dynamics, establishing state-of-the-art spatiotemporal consistency, yielding consistent user preference (39% relative improvement in content preservation and 12% in temporal consistency), and achieving a significant 5% gain in downstream road line segmentation and 7% gain in downstream multi-object detection.