Video World Models · 2026-10-06
EnvDreamer: Large-Scale Multimodal-to-Environment Generation for Embodied AI
Kabir Swain, Sijie Han, Antonio Torralba
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
EnvDreamer is a language-first framework that generates validator-passed, physics-enabled Unreal Engine 5 environments for embodied AI, world models, and robotics. It plans assets/layout/task logic, compiles to Unreal Python, applies edits via Remote Control, and validates playability, physics consistency, and goal completeness. The authors release EnvDreamer-20k (20,000 environments) and show competitive zero-shot/few-shot results after pretraining on multiple embodied benchmarks.
Why it matters
The authors aim to scale embodied environments to tens of thousands of interactive scenes with reproducibility and executable task programs. EnvDreamer provides language-driven environment generation, unit-style validators to ensure playability/physics/goal completeness, and a released dataset for world-model pretraining and embodied policy learning across navigation, rearrangement, and manipulation.
Method
- Plan–compile–apply–validate loop: decompose prompts/images into scene program/layout and task specifications in JSON, compile to Unreal Python, apply via UE Remote Control, and validate with unit-style checks.
- Generate diverse interactive UE5 environments with a large asset library, deterministic seeds, and per-load variation in textures/materials/lighting; tasks compile to executable programs with preconditions and success predicates.
- Release EnvDreamer-20k: 20,000 validator-passed environments including task programs, scene graphs, trajectories, and metadata; also support image-conditioned reconstruction for real-to-sim.
Limitation
From the Discussion and Limitations section, the paper discusses scope (indoor only), task complexity (short horizons, single solutions), and generation latency (2–5 minutes per scene), plus CPU-bound throughput when scaling parallel instances.
Abstract (from arXiv)
Large datasets and high capacity models have accelerated progress in vision and language. This work introduces a platform aimed at bringing comparable gains to embodied learning, world models, and robotics. We present EnvDreamer, a framework that uses large language and vision language models to generate Unreal Engine 5 environments for embodied AI and robot training. EnvDreamer enables sampling of large, diverse, interactive, customizable, and validator passed virtual environments for training and evaluation across navigation, interaction, and manipulation. We illustrate the platform with a large set of generated scenes and simple baselines. Policies trained on EnvDreamer generated environments, without explicit mapping or human task supervision, achieve competitive results on multiple embodied benchmarks spanning navigation, rearrangement, and manipulation. EnvDreamer also supports image-conditioned reconstruction for real-to-sim studies. Finally, we release EnvDreamer-20k, a dataset of 20,000 validator passed environments with task programs, scene graphs, trajectories, and metadata to support reproducible benchmarking.
Related papers
- EmbodiedWorld-200K: Toward Embodied World Agents via Embodied-Planning Dataset and Interactive World Models
- Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
- RynnUnify: Towards Unified Embodied Brain
- Learning Interactive Real-World Simulators
- Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC
EnvDreamer: Large-Scale Multimodal-to-Environment Generation for Embodied AI
Preprint posted: 2026-10-03 · Original record ↗
- arxiv
- 2610.04301
Explicit source versions (1)
- v1 · 2026-10-03
- Preprint available
Source date: 2026-10-03 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Large datasets and high capacity models have accelerated progress in vision and language. This work introduces a platform aimed at bringing comparable gains to embodied learning, world models, and robotics. We present EnvDreamer, a framework that uses large language and vision language models to generate Unreal Engine 5 environments for embodied AI and robot training. EnvDreamer enables sampling of large, diverse, interactive, customizable, and validator passed virtual environments for training and evaluation across navigation, interaction, and manipulation. We illustrate the platform with a large set of generated scenes and simple baselines. Policies trained on EnvDreamer generated environments, without explicit mapping or human task supervision, achieve competitive results on multiple embodied benchmarks spanning navigation, rearrangement, and manipulation. EnvDreamer also supports image-conditioned reconstruction for real-to-sim studies. Finally, we release EnvDreamer-20k, a dataset of 20,000 validator passed environments with task programs, scene graphs, trajectories, and metadata to support reproducible benchmarking.
