Latent Dynamics · 2026-11-05
SurgTransfer: Multi-Step Latent Action Modeling for Action Transfer in Surgical World Model
Saurabh Koju, Love Panta, Deepak Thapa, Rudra P. K. Poudel, Prajwal Ghimire, Angus Watson, Yash Raj Shrestha, Prashnna Kumar Gyawali, Binod Bhattarai
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
SurgTransfer learns transferable latent surgical actions from unlabeled videos. It introduces a multi-step latent action model (LAM) that learns a single composite action spanning longer horizons, and an action-conditioned multi-step dynamics model based on finetuning Stable Video Diffusion. The authors test on SurgToolLoc-2022 and PitVis-2023 and evaluate with quantitative metrics and qualitative assessments by surgical experts.
Why it matters
The authors argue that prior surgical latent-action world models are “myopic” because they infer only actions between consecutive frames, while surgery is hierarchical and goal-directed. SurgTransfer aims to learn context-invariant, composable latent actions for action transfer across surgical contexts, with publicly available datasets and reported quantitative/qualitative evaluations.
Method
- Introduces a general surgical procedure transfer framework (SurgTransfer) that transfers learned latent actions from a source video to a target context.
- Proposes multi-step LAM: predicts multiple future frames from an initial context frame, learning composite latent actions spanning longer time windows.
- Finetunes action-controllable Stable Video Diffusion for multi-step, action-conditioned video prediction using a composite action representation.
Abstract (from arXiv)
Surgical world models have the potential to provide intra-operative guidance, accelerate risk-free training and facilitate transfer of expert surgical maneuvers. However, the use of world models for transferring surgical expertise has been largely unexplored. Moreover, recent approaches in this domain have mainly relied on ground truth kinematics actions or expert annotations for specific tasks, limiting their applicability beyond the narrow range of tasks with ground-truth annotations. In contrast, existing works in the natural domain have shown promising results in learning latent actions solely from unlabeled videos and transferring the learned actions across contexts. We observe that this paradigm is short-sighted for the surgical domain, failing to capture the hierarchical and goal-directed structure of surgical procedures. Thus, we propose a novel multi-step action learning framework to learn context-invariant latent actions that are easily transferable across various surgical contexts. We perform extensive experiments on the publicly available datasets, SurgToolLoc-2022 and PitVis-2023, verifying our approach with both quantitative metrics and qualitative evaluation by surgical experts. We have published our code and the implementation details at https://github.com/bhattarailab/Surgical-Vision-World-Model.
Related papers
- SurgWorld: Learning Surgical Representations Beyond Observation via Latent Prediction
- SurgTAP-3D: A Large-Scale Cross-Surgery Video Dataset with Contact-Aware 3D Motion Tracking
- SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models
- RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
- W2A: Distilling World Priors into Action Models for Real-Time Robotic Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-05 · Indexed by Pub-AI: 2026-11-05 UTC
SurgTransfer: Multi-Step Latent Action Modeling for Action Transfer in Surgical World Model
First announced by arXiv: 2026-11-05 · Original record ↗
- arxiv
- 2611.05443
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-05 · First detected: 2026-11-05 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.05443v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Surgical world models have the potential to provide intra-operative guidance, accelerate risk-free training and facilitate transfer of expert surgical maneuvers. However, the use of world models for transferring surgical expertise has been largely unexplored. Moreover, recent approaches in this domain have mainly relied on ground truth kinematics actions or expert annotations for specific tasks, limiting their applicability beyond the narrow range of tasks with ground-truth annotations. In contrast, existing works in the natural domain have shown promising results in learning latent actions solely from unlabeled videos and transferring the learned actions across contexts. We observe that this paradigm is short-sighted for the surgical domain, failing to capture the hierarchical and goal-directed structure of surgical procedures. Thus, we propose a novel multi-step action learning framework to learn context-invariant latent actions that are easily transferable across various surgical contexts. We perform extensive experiments on the publicly available datasets, SurgToolLoc-2022 and PitVis-2023, verifying our approach with both quantitative metrics and qualitative evaluation by surgical experts. We have published our code and the implementation details at https://github.com/bhattarailab/Surgical-Vision-World-Model.
