Video World Models · 2026-10-01

Audible World Models: Spatially Aware Sound Generation for 3D Worlds

Duowen Chen, Jinjin He, Gouthaman KV, Sandeep Bangalore Venkatesh, Bo Zhu

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Audible World Models is a training-free pipeline that turns a text prompt into an “audible world” state (semantic sound labels, dry audio assets, 3D source placements, source attributes, scene geometry, acoustic parameters). It then renders listener-dependent binaural audio via geometric acoustic propagation so sound sources remain persistent while audio changes with viewpoint and motion.

Why it matters

The authors address that generative 3D worlds are often silent or use soundtracks without an explicit representation of where sources are located and how sound varies as the listener moves. By storing an explicit, reusable sound field tied to reconstructed geometry, the same generated world can be heard from different trajectories with more spatially consistent, motion-dependent audio.

Method

From a text prompt, build a panoramic 3D proxy, decompose it into foreground/background/sky layers, and use VLM agents to produce sound labels with locality attributes and segmentation support.
Synthesize dry audio clips per sound label (using Stable Audio 1.0 in the current implementation), then ground them as persistent 3D sources using reconstructed geometry (including localized foreground and diffuse background sources).
Predict acoustic parameters with an agent and render binaural audio along the listener trajectory using GSound geometric acoustic propagation.

Limitation

The paper notes that the evaluated pipeline is offline and that GSound rendering is CPU-based; it also states that results include releases of baseline configurations with shorter 5–10 s outputs, requiring duration-matched comparisons rather than mixing durations.

Abstract (from arXiv)

Text- and image-conditioned world generators can create visually rich 3D environments, yet these worlds often remain silent or rely on soundtracks synthesized solely from text or rendered video. Although such audio can convey what should be heard, it lacks an explicit representation of where sound sources are located and how their perceived sound should vary with listener movement. We introduce Audible World Models, a training-free framework that incorporates sound into the generated world state. Starting from a text prompt, our system constructs a panoramic 3D proxy, separates it into semantic layers, identifies sound-producing foreground objects and ambient background regions, and synthesizes dry audio for each sound label. It then anchors these sources to reconstructed geometry and renders listener-dependent spatial audio using geometric acoustic propagation. By explicitly linking semantics, geometry, and sound propagation, the framework maintains persistent source locations while adapting the rendered audio to changes in listener viewpoint and motion. Experiments across 80 generated scenes demonstrate substantial gains in spatial consistency over text-, video-, and panorama-conditioned baselines, while preserving competitive semantic alignment. VLM-based assessments and human evaluations further indicate that our soundtracks are preferred for their audio-visual consistency, spatial plausibility, and motion-dependent behavior.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-29 · Indexed by Pub-AI: 2026-10-01 UTC

preprint