Driving · 2026-11-13

BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model

Junyoung Seo, Jaewoo Seo, Seungryong Kim, Laura Leal-Taix\'e, Xiuming Zhang, Seung Wook Kim

arXiv:2611.14815PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

BirdDreams is an autoregressive driving world model that jointly generates perspective video and a schematic metric bird’s-eye-view (BEV) video. It trains on synchronized RGB–BEV pairs rendered from annotations and uses the predicted BEV as a geometric interface to condition ego-trajectory rollouts without HD map or vehicle-state inputs at inference.

Why it matters

Learning reactive driving from perspective-only video is hard due to entangled motion/appearance and occlusions. BirdDreams makes scene layout part of the prediction by jointly generating a metric BEV (simple shapes) and photorealistic perspective video, enabling ego-trajectory following via BEV reprojection and improving realism of surrounding-vehicle behavior.

Method

  • Joint prediction: two video diffusion-transformer branches (perspective and schematic BEV) with joint attention during denoising for synchronized generation.
  • Metric BEV supervision: BEV targets are rendered from scene annotations as simple shapes at a fixed metric scale with reduced occlusion.
  • BEV-aware ego conditioning: reproject the latest predicted BEV frame along an input ego trajectory (translate/rotate in BEV space) as a soft condition for rollout.
Abstract (from arXiv)

Driving world models aim to simulate how a scene unfolds under different ego actions. This requires plausible motion of surrounding vehicles as well as precise ego-trajectory control. Learning vehicle motion from perspective video alone is challenging because motion is entangled with appearance and vehicles can be occluded. Prior work therefore conditions video generation on layouts that specify vehicle motion, providing control over surrounding vehicles but requiring their future trajectories as inputs. We introduce BirdDreams, an autoregressive driving world model that jointly generates perspective video and a schematic bird's-eye-view (BEV) video, making scene layout part of the predicted future. The BEV, rendered from scene annotations for training, depicts roads and vehicles as simple shapes at a fixed metric scale, providing a top-down view of their positions and motion with reduced occlusion between vehicles. Perspective video, by contrast, retains scene details that the BEV omits, such as traffic signals and object appearance. Joint attention between two video diffusion transformers, one per stream, lets each stream inform the other during generation. The predicted BEV also provides a geometric interface for ego control: we translate and rotate its latest frame along an input ego trajectory and use the resulting frames as a soft condition that shows how the scene would appear to move if only the ego vehicle moved. At inference, BirdDreams generates both videos from a single camera image without requiring an HD map or vehicle-state inputs. On PhysicalAI-AV and nuScenes, BirdDreams follows input ego trajectories more faithfully than prior driving world models and generates more plausible ego motion without ego-action conditioning. Human evaluation shows that BirdDreams outperforms a perspective-only baseline in generating realistic surrounding-vehicle behavior.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model

Junyoung Seo, Jaewoo Seo, Seungryong Kim, Laura Leal-Taixé, Xiuming Zhang, Seung Wook Kim

First announced by arXiv: 2026-11-13 · Original record ↗

arxiv
2611.14815
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.14815v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Driving world models aim to simulate how a scene unfolds under different ego actions. This requires plausible motion of surrounding vehicles as well as precise ego-trajectory control. Learning vehicle motion from perspective video alone is challenging because motion is entangled with appearance and vehicles can be occluded. Prior work therefore conditions video generation on layouts that specify vehicle motion, providing control over surrounding vehicles but requiring their future trajectories as inputs. We introduce BirdDreams, an autoregressive driving world model that jointly generates perspective video and a schematic bird's-eye-view (BEV) video, making scene layout part of the predicted future. The BEV, rendered from scene annotations for training, depicts roads and vehicles as simple shapes at a fixed metric scale, providing a top-down view of their positions and motion with reduced occlusion between vehicles. Perspective video, by contrast, retains scene details that the BEV omits, such as traffic signals and object appearance. Joint attention between two video diffusion transformers, one per stream, lets each stream inform the other during generation. The predicted BEV also provides a geometric interface for ego control: we translate and rotate its latest frame along an input ego trajectory and use the resulting frames as a soft condition that shows how the scene would appear to move if only the ego vehicle moved. At inference, BirdDreams generates both videos from a single camera image without requiring an HD map or vehicle-state inputs. On PhysicalAI-AV and nuScenes, BirdDreams follows input ego trajectories more faithfully than prior driving world models and generates more plausible ego motion without ego-action conditioning. Human evaluation shows that BirdDreams outperforms a perspective-only baseline in generating realistic surrounding-vehicle behavior.