Video World Models · 2026-10-07

WorldSonus: Bringing Sound to Worlds

Pengjun Fang, Jingyi Fa, Kam Man Wu, Jiaming Wang, Haoyuan Huang, Yaguang Wu, Xiangjun Huang, Ziyang Ma, Weijia Chen, Hongyu Liu, Zeyue Tian, Qifeng Chen

arXiv:2610.08760 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

WorldSonus is an interactive video-to-audio framework for real-time spatial stereo in world models. It uses streaming causal autoregressive diffusion to synthesize 100ms audio chunks at RTF 0.41, supports mid-stream text prompt updates via chunk-indexed prompt scheduling, and trains spatially aligned stereo using clear-stereo and panoramic FOA supervision. Experiments show it matches or outperforms bidirectional baselines on acoustic quality and stereo balance.

Why it matters

World models are increasingly capable at visual synthesis but remain largely silent. WorldSonus targets three requirements for interactive use: causal real-time audio generation, mid-stream controllability from updated sound instructions, and camera-aligned spatial stereo. The paper evaluates on clear-stereo VGGSound and an Interactive benchmark (gameplay and real-world footage) and reports multiple audio quality, alignment, and stereo-balance metrics, including prompt-switch control and long-horizon stability.

Method

Streaming causal autoregressive diffusion: decoder-only Transformer with bounded Ring-KV cache (W=50 chunks), synthesizing 100ms stereo audio chunks at 30 FPS latent rate with a rectified-flow objective.
Mid-stream interactive control: text encoded by T5Gemma 2 and inserted via cross-attention cache replacement at nearest chunk boundary; trained with time-varying prompt schedules (switch, drop, arrival).
Spatial alignment: two-timescale visual conditioning (chunk summaries + frame-aligned local tokens) and ShiftNCE training with a frozen synchronization teacher; stereo supervision from clear-stereo and panoramic FOA data.

Abstract (from arXiv)

Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Project page: https://noizai.github.io/WorldSonus/

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC

preprint