Video World Models · 2026-10-15

SonoWorld: Towards Interactive Pose-Controllable Ultrasound World Models

Zefan Yang, Zheyuan Zhang, Boris Maihe, Yikang Liu, Hang Yu, Ankush Mukherjee, Xiao Chen, Pingkun Yan, Terrence Chen, Shanhui Sun

arXiv:2610.19327 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SonoWorld is a causal ultrasound world model trained primarily from unlabeled freehand clinical ultrasound videos. It learns latent-action dynamics from ~6.9M multi-organ frames, then grounds them to 6-DoF probe control using a small EM-tracked thyroid dataset, and distills a bidirectional diffusion model into a causal long-horizon generator with selective KV-cache retention for interactive rollouts.

Why it matters

The authors aim to enable safe, reusable, pose-controllable simulation for robotic ultrasound data acquisition and policy evaluation. Because ultrasound views are sensitive to small probe motions, the work focuses on directly generating future ultrasound observations conditioned on commanded 6-DoF probe actions, rather than only predicting latent spaces or relying on tracked poses at inference.

Method

Pretrain a latent-action-conditioned video diffusion model on ~6.9M unlabeled multi-organ ultrasound frames to learn dynamics without pose annotations.
Post-train with EM-tracked thyroid scans to replace latent actions with metric 6-DoF relative probe motions (actions in the probe frame).
Distill to a causal autoregressive generator using self-forcing and selectively retain high-attention KV-cache tokens for streaming, long-horizon action-conditioned generation.

Limitation

The authors note that surface-normal translation is physically constrained by probe–skin contact and tissue compression, so freehand scanning explores a contact-constrained subset of the 6-DoF pose space.

Abstract (from arXiv)

Robotic ultrasound can standardize image acquisition and reduce operator workload and variability, but policy learning remains limited by the lack of realistic, controllable simulation environments. While controllable world models have advanced rapidly in natural-image domains, ultrasound world modeling remains challenging because each frame captures only a cross-section of unobserved 3D anatomy, causing small probe motions to induce large appearance changes. Existing approaches rely on robot-tracked poses and latent-space prediction, but do not directly generate the observations required for interactive control. We propose SonoWorld, a causal ultrasound world model that learns video dynamics from unlabeled clinical videos and grounds them to 6-DoF control using a small set of freehand clinical scans tracked with an EM sensor. SonoWorld is trained in three stages: pretraining a latent-action-conditioned video diffusion model on around 6.9 million unlabeled multi-organ ultrasound frames; grounding the learned video dynamics to metric probe poses with a small EM-tracked thyroid dataset; and applying distribution matching distillation to enable causal, long-horizon autoregressive generation with selectively caching eviction tokens at inference time. We evaluate controllability through pose-following fidelity, round-trip consistency, and axis-specific control under synthetic actions on held-out tracked scans, and further validate it with a user study. In model-based action discrimination, SonoWorld correctly ranks the target-directed probe motion over an equal-magnitude opposite motion in up to 74.8% of trials. To our knowledge, SonoWorld is the first causal, pose-controllable ultrasound world model learned primarily from freehand clinical videos, providing a data-driven interactive environment for robotic ultrasound data acquisition and policy evaluation.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC

preprint