Latent Dynamics · 2026-11-10
SurgWorld: Learning Surgical Representations Beyond Observation via Latent Prediction
Jingkun Yue, Pengxiang Wang, Siqi Zhang, Zinan Jia, Yiyao Zhang, Huihuan Xu, Yichang Hao, Xiaohong Liu, Zongbo Han, Guangyu Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
SurgWorld is a video-native, self-supervised framework for surgical representation learning beyond visible observations. It treats a relative spatio-temporal shift as a pseudo-action and uses Out-of-Sight Dynamics Modeling (OSDM) to predict the target latent representation from a partial context. A Geometric Visual Similarity (GVS) loss aligns patch-level relational structures with a frozen teacher. An agglomerative training framework integrates multiple teachers to learn unified latent dynamics and geometry.
Why it matters
The authors aim to address partial observability in surgery by explicitly modeling out-of-sight surgical dynamics via latent prediction rather than only reconstructing observed content or predicting future latents within the current field of view. The paper reports transfer learning results across 17 surgical benchmarks and includes analyses intended to show the model captures dynamics beyond directly observed context.
Method
- Out-of-Sight Dynamics Modeling (OSDM): sample context and target clips, represent their relative spatio-temporal shift as a pseudo-action, and predict target latent representation from partial context under an EMA teacher supervision.
- Geometric Visual Similarity (GVS): align patch-level pairwise visual similarity structures between predicted/student representations and a frozen dense-feature teacher to preserve spatial geometry and local visual coherence.
- Unified agglomerative training: integrate OSDM and GVS (and complementary teacher signals) within a single forward pass using shared contexts, targets, and actions; GVS weight λ_GVS is set to 2.0.
Abstract (from arXiv)
Surgical procedures are inherently partially observable, requiring surgeons to continuously infer evolving anatomical dynamics and future surgical states beyond the current observation. However, existing self-supervised learning methods primarily learn from observed surgical videos through masked reconstruction or predicting representations within the camera's current field of view, lacking the capability to explicitly model the out-of-sight surgical dynamics. Inspired by latent predictive world models, we introduce SurgWorld, a video-native, self-supervised framework for surgical representation learning, which treats a specified spatial or temporal shift as a pseudo-action and learns to predict the target latent representation from a partial observation. Specifically, we propose Out-of-Sight Dynamics Modeling (OSDM) as the pretraining objective for predicting representations conditioned on relative spatio-temporal shifts. We further regularize the learned latent dynamics from OSDM with a Geometric Visual Similarity (GVS) objective by aligning patch-level relational structures between predicted and target representations, preserving spatial geometry and local visual coherence for dense perception. These two objectives are integrated through an agglomerative training framework that consolidates complementary supervisory signals from multiple teachers to learn a unified latent representation of surgical dynamics and visual geometry. Extensive transfer learning experiments on 17 surgical benchmarks show that SurgWorld consistently outperforms existing vision foundation models across surgical workflow recognition, fine-grained action understanding, and dense visual perception. We further conduct tailored qualitative and quantitative analyses, demonstrating that SurgWorld effectively captures surgical dynamics beyond the directly observed context through latent prediction.
Related papers
- SurgTransfer: Multi-Step Latent Action Modeling for Action Transfer in Surgical World Model
- SurgTAP-3D: A Large-Scale Cross-Surgery Video Dataset with Contact-Aware 3D Motion Tracking
- EndoWorld-JEPA: Motion-Guided Global-Local Latent World Modeling for Endoscopic Video Representation Learning
- SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models
- ForeAct: Geometry-Guided Action Learning from World Representations
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
SurgWorld: Learning Surgical Representations Beyond Observation via Latent Prediction
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.11140
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11140v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Surgical procedures are inherently partially observable, requiring surgeons to continuously infer evolving anatomical dynamics and future surgical states beyond the current observation. However, existing self-supervised learning methods primarily learn from observed surgical videos through masked reconstruction or predicting representations within the camera's current field of view, lacking the capability to explicitly model the out-of-sight surgical dynamics. Inspired by latent predictive world models, we introduce SurgWorld, a video-native, self-supervised framework for surgical representation learning, which treats a specified spatial or temporal shift as a pseudo-action and learns to predict the target latent representation from a partial observation. Specifically, we propose Out-of-Sight Dynamics Modeling (OSDM) as the pretraining objective for predicting representations conditioned on relative spatio-temporal shifts. We further regularize the learned latent dynamics from OSDM with a Geometric Visual Similarity (GVS) objective by aligning patch-level relational structures between predicted and target representations, preserving spatial geometry and local visual coherence for dense perception. These two objectives are integrated through an agglomerative training framework that consolidates complementary supervisory signals from multiple teachers to learn a unified latent representation of surgical dynamics and visual geometry. Extensive transfer learning experiments on 17 surgical benchmarks show that SurgWorld consistently outperforms existing vision foundation models across surgical workflow recognition, fine-grained action understanding, and dense visual perception. We further conduct tailored qualitative and quantitative analyses, demonstrating that SurgWorld effectively captures surgical dynamics beyond the directly observed context through latent prediction.
