Video World Models · 2026-11-02
Dreams Are Cheaper Than Flights: Curated World-Model Data Improves Aerial Detection
Rugved Katole, Vedant Patil, Jenna Kline, Christopher Stewart
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors seed a 7B video world model (Cosmos) with drone wildlife frames, then use an expert to reject deformed generations and annotate accepted frames. On held-out field videos from the same drones, mAP@50–95 rises 4.3% when curated synthetic frames are added to real data and 10.3% when they replace it; on an urban benchmark it reaches +33.5%. They also estimate synthetic generation is cheaper than field labor.
Why it matters
Field data collection is slow, costly, and disruptive for aerial wildlife/urban detection. The paper tests whether seeded, curated synthetic data can match real field data for training detectors evaluated on held-out field videos, and reports both detection gains and a compute-vs-field-labor cost comparison. It highlights that even with high rejection rates, compute-based synthetic data can be a practical substitute when transfer is verified and the output is curated.
Method
- Field-to-simulation-to-field protocol: seed a world model only with training footage from the same drones, then have an expert curate generated clips and train detectors using real vs curated synthetic as the only variable.
- Generate with Cosmos-Predict1-7B (Video2World): each clip is produced from nine consecutive training frames plus a short text prompt; accept clips only if a domain expert finds no deformities.
- Evaluation: fine-tune YOLO detectors (YOLO11m on MMLA, YOLOv8m on VisDrone) and report mAP@50–95 on held-out field test/validation splits from the same drones.
Limitation
Synthetic clips must be curated and transfer verified per class; the authors note world models can produce deformities and poorly rendered classes can vanish in aggregate accuracy (e.g., bus on VisDrone).
Abstract (from arXiv)
Robot perception is increasingly data driven, yet field data is slow, costly, and sometimes disruptive to collect. Video world models offer a cheaper source: seeded with real frames, they extend a recorded scene into new footage with the deployment's sensor, altitude, and environment. We ask whether such synthetic data matches the utility of real field data and whether detectors trained with it perform in the field. Seeding a 7B-parameter world model with wildlife drone video, we more than double the training set with curated frames. On held-out videos from the same drones, mAP@50--95 rises 4.3\% when these frames are added to real data and 10.3\% when they replace it, and up to 33.5\% on an urban benchmark. Although an expert rejects an estimated 87.5\% of generated clips, the data costs about \$1,100 of GPU time, against more than \$3,200 of field labor alone for a less useful real training set. Synthetic data thus shifts data costs from fieldwork to compute.
Related papers
- ScanDreamer: A Map-Grounded World Model for Quadrotor Navigation
- SearchWorld: Spatial Value-Grounded Imagination for UAV Object Search via World Models
- From Video to Digital Twin: An Exploratory Study of World Models and Gaussian Splatting on In-the-Wild Video
- WanderBench: Can World Models Explore Dynamic Worlds from Video?
- Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-02 · Indexed by Pub-AI: 2026-11-02 UTC
Dreams Are Cheaper Than Flights: Curated World-Model Data Improves Aerial Detection
First announced by arXiv: 2026-11-02 · Original record ↗
- arxiv
- 2610.41030
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-02 · First detected: 2026-11-02 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.41030v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Robot perception is increasingly data driven, yet field data is slow, costly, and sometimes disruptive to collect. Video world models offer a cheaper source: seeded with real frames, they extend a recorded scene into new footage with the deployment's sensor, altitude, and environment. We ask whether such synthetic data matches the utility of real field data and whether detectors trained with it perform in the field. Seeding a 7B-parameter world model with wildlife drone video, we more than double the training set with curated frames. On held-out videos from the same drones, mAP@50--95 rises 4.3\% when these frames are added to real data and 10.3\% when they replace it, and up to 33.5\% on an urban benchmark. Although an expert rejects an estimated 87.5\% of generated clips, the data costs about \$1,100 of GPU time, against more than \$3,200 of field labor alone for a less useful real training set. Synthetic data thus shifts data costs from fieldwork to compute.
