Driving · 2026-10-14

SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models

Wanhao Liu, Rulin Zhou, Liangjing Shao, Zhaocheng Lin, Dongyue Li, Jinsong Lin, Guoheng Ma, Hangheng Yang, Zhiqing Tang, Jing Chen, Panshuo Li, Hongliang Ren

arXiv:2610.15645 PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SurgCast predicts future instrument skeleton keypoints from current structure, robot state, and future actions, then uses the forecast to control action-conditioned surgical video generation. Dual-path injection injects (1) rasterized skeleton geometry via geometric residuals and (2) derived instrument kinematic semantics via conditioning modulation, alongside native action conditioning. On SutureBot and SRTH-Porcine-Cholei, predicted-skeleton teacher and four-step causal student improve quality and instrument-consistency metrics, including FVD decreases.

Why it matters

The authors propose an explicit control interface for surgical world models: future articulated instrument structure is forecast from prospective robot actions and then used to guide video generation. This targets failures of direct action conditioning (instrument drift and misalignment) by adding instrument-meaningful structure and motion semantics to the generation process. Reported improvements across multiple quality and tool-consistency metrics support using predicted skeleton geometry and kinematic semantics for controllable surgical video prediction.

Method

Action-to-future-skeleton predictor: forecasts future instrument skeleton keypoints and visibility from current completed structure, robot state, and prospective actions using a causally masked transformer with coordinate/visibility-aware losses.
Dual-path semantic–kinematic injection: converts the forecast into (i) multi-scale geometric residuals from rasterized keypoints/edges and (ii) global/local kinematic descriptors (translation, shaft-direction angular velocity, wrist/jaw articulation) for conditioning modulation; both complement native action inputs.
Teacher + causal student: trains a full-sequence teacher with rectified-flow objective and distills into a four-step causal student using distribution matching and bidirectional-to-causal distillation while retaining the same structural interface.

Limitation

The evaluation uses supplied action windows; sustained closed-loop robot control remains to be validated. Testing the interface under continuous feedback is the next step toward robotic decision-making use.

Abstract (from arXiv)

Surgical world models must turn prospective robot actions into visually faithful, structurally consistent instrument motion. We introduce SurgCast, which predicts future instrument skeletons from current structure, robot state, and future actions, then uses the forecast to control video generation. Dual-path injection combines skeleton geometry with derived, instrument-specific motion descriptors through geometric residuals and conditioning modulation. Both paths complement native action conditioning and are retained in a four-step causal student. On SutureBot and SRTH-Porcine-Cholei, the teacher and student improve all seven reported quality and instrument-consistency metrics over their respective direct-action baselines; teacher FVD decreases by 37.6% and 67.3%. Ablations show that future actions improve skeleton prediction and that combining both injection paths performs best across all six metrics under predicted skeletons. These results support using predicted articulated structure and its motion semantics as a control interface for surgical video prediction. Project videos and examples are available at https://surgcast.netlify.app/.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-11 · Indexed by Pub-AI: 2026-10-14 UTC

preprint