Video World Models · 2026-10-08

UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map

Keke Yang, Erqi Wang, Sainan Guan, Hongliang Ren

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

UltraWorld learns interactive ultrasound world models from untracked clinical videos without real action annotations. It adapts a video foundation model into a mask-conditioned ultrasound generator, then uses 3D anatomical assets and programmable probe trajectories to synthesize action–video pairs for self-distillation into an observation+action world model. It further introduces the Acoustic Sampling Map (AsMap) to encode pixel-wise 3D sampling geometry for improved action following.

Why it matters

The authors target a practical gap: clinical ultrasound video is abundant but often lacks synchronized probe motion, and ultrasound actions change what cross-sections are sampled. UltraWorld aims to transfer visual/temporal priors from action-free clinical recordings into action-conditioned prediction and planning, using programmable anatomy-derived supervision and a sampling-geometry representation (AsMap).

Method

Learn a controllable ultrasound video generator from action-free clinical ultrasound videos using reference images and temporally aligned anatomical masks; no acquisition states are required for this stage.
Construct missing action supervision by sampling masks along programmable acquisition trajectories in 3D anatomy, yielding synthetic action–video pairs for self-distillation.
Introduce Acoustic Sampling Map (AsMap), a pixel-aligned representation of probe pose and ultrasound imaging settings (sampling positions, beam directions, depths), and inject it as conditioning into an action-conditioned ultrasound world model.

Limitation

UltraWorld is trained primarily with generator-synthesized action–video trajectories rather than real ultrasound sequences with synchronized probe tracking, so errors or biases in the generative prior may propagate to the distilled world model. AsMap represents acquisition geometry but does not explicitly model tissue-dependent acoustic effects such as scattering, attenuation, and speckle formation. Evaluation on real clinical ultrasound is limited to stability and round-trip consistency because paired future observations with known actions are unavailable, and closed-loop robotic planning is validated only in simulation and on a limited number of episodes.

Abstract (from arXiv)

World models can enable autonomous ultrasound scanning by predicting the outcomes of probe motions from local observations. Learning this action--observation relationship typically relies on synchronized video--pose pairs, which are costly to collect at scale and largely unavailable in routine clinical recordings. Reliable action following further requires modeling ultrasound's cross-sectional sampling geometry. We present UltraWorld, a self-distillation recipe that transfers priors from clinical ultrasound videos into interactive world models without real action annotations. Starting from clinical videos, we adapt a video foundation model into an ultrasound generator conditioned on reference images and anatomical masks. Anatomical masks sampled along programmable trajectories through 3D anatomy provide spatial guidance for synthesizing action--video pairs. We then use these synthetic pairs to self-distill the generator into a world model that predicts future observations from local observations and actions, without requiring anatomical masks or other 3D assets at inference time. To further improve action following, we introduce the Acoustic Sampling Map (AsMap), which represents probe poses and imaging settings as pixel-wise 3D sampling positions, beam directions, and depths. Experiments demonstrate improved prediction fidelity and action following. Across nine simulated closed-loop local planning episodes, UltraWorld reduces the mean final distance to the goal and orientation error by 29\% and 38\%, respectively, compared with visual servoing. Project Page: https://ultraworld-project.github.io/.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC

preprint