Video World Models · 2026-11-10
SurgTAP-3D: A Large-Scale Cross-Surgery Video Dataset with Contact-Aware 3D Motion Tracking
Jiaxin Guo, Wenzhen Dong, Taoyu Wu, Haomin Kuang, Mengya Xu, Kejian Shi, Ziyi Wang, Tongfan Guan, Mingkai Jia, Yue Wang, Qi Dou, Yun-Hui Liu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
SurgTAP-3D is a large-scale in-vivo pseudo-3D surgical motion tracking dataset and benchmark for contact-aware 3D point tracking in minimally invasive surgery. It includes 520K frames, 4,549 action clips, and 1.4M 2D/3D point trajectories from diverse robotic and manual procedures under unified action categories. The authors propose a unified framework to convert uncalibrated monocular surgical videos into structured 4D annotations via contact-region prompting, active-tool segmentation, dynamic bundle adjustment, and global 4D refinement.
Why it matters
The paper addresses limits of existing surgical datasets (largely ex-vivo/simulated, limited scale/diversity, and lack of contact-aware tool–tissue interaction annotations). SurgTAP-3D aims to provide large-scale, contact-aware 3D supervision to support downstream tasks such as 3D surgical point tracking, instrument displacement estimation, deformable surface reconstruction, and foundation-model fine-tuning, and provides a benchmark across SurgPose, SurgT, and STIR for both instrument and tissue tracking.
Method
- Unified 3D contact-aware point tracking framework that converts uncalibrated monocular surgical videos into structured 4D annotations using contact-region prompting, active-tool segmentation, dynamic bundle adjustment, and global 4D refinement.
- Pipeline uses priors from feed-forward foundation models for reconstruction, point tracking, and segmentation to produce temporally consistent trajectories, pseudo-scale geometry, camera poses, and instrument masks.
- Global 4D optimization refines geometry and point trajectories by enforcing constraints of instrument rigidity, articulated flow consistency, and temporal smoothness.
Limitation
Under monocular reconstruction, global coordinates and pseudo-scale geometry remain pseudo-scale unless additional metric calibration is available.
Abstract (from arXiv)
Understanding 3D scene dynamics in minimally invasive surgery (MIS) is a fundamental step toward physically intelligent and autonomous surgical systems. However, existing public datasets offer limited scalability and diversity in real-world clinical scenarios. Moreover, annotations typically focus on either tissue or instruments alone, overlooking tool-tissue interactions. This gap limits applications such as augmented reality guidance and autonomous surgery driven by vision-language-action (VLA) models or world models, which demand large-scale, high-quality 3D supervision. In this work, we construct SurgTAP-3D, a large-scale in-vivo pseudo-3D surgical motion tracking dataset and benchmark, comprising 520K frames, 4,549 action clips, and 1.4M 2D/3D point trajectories from diverse robotic and manual procedures under unified action categories. To make annotation scalable, we design a unified framework for surgical 3D point tracking that converts uncalibrated monocular surgical videos into structured 4D annotations through contact-region prompting, active-tool segmentation, dynamic bundle adjustment, and global 4D refinement. This pipeline adapts foundation priors to surgical videos and produces temporally consistent trajectories, pseudo-scale geometry, camera poses, and instrument masks. The dataset captures realistic challenges inherent to surgical environments, including occlusions, smoke, bleeding, and rapid instrument motion. We establish a comprehensive benchmark on SurgPose, SurgT, and STIR, covering both instrument and tissue tracking. We demonstrate instrument displacement estimation, deformable surface reconstruction, and foundation-model fine-tuning. Extensive experiments and analysis demonstrate the effectiveness of our dataset and annotation framework for 3D surgical point tracking, reconstruction, and surgical action understanding. Homepage: https://wrld.github.io/surgtap-3d/
Related papers
- SurgWorld: Learning Surgical Representations Beyond Observation via Latent Prediction
- SurgTransfer: Multi-Step Latent Action Modeling for Action Transfer in Surgical World Model
- SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models
- SonoWorld: Towards Interactive Pose-Controllable Ultrasound World Models
- Observation-Referenced Tissue Deformation Forecasting with Tool-Driven and Residual Dynamics
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
SurgTAP-3D: A Large-Scale Cross-Surgery Video Dataset with Contact-Aware 3D Motion Tracking
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.11697
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11697v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Understanding 3D scene dynamics in minimally invasive surgery (MIS) is a fundamental step toward physically intelligent and autonomous surgical systems. However, existing public datasets offer limited scalability and diversity in real-world clinical scenarios. Moreover, annotations typically focus on either tissue or instruments alone, overlooking tool-tissue interactions. This gap limits applications such as augmented reality guidance and autonomous surgery driven by vision-language-action (VLA) models or world models, which demand large-scale, high-quality 3D supervision. In this work, we construct SurgTAP-3D, a large-scale in-vivo pseudo-3D surgical motion tracking dataset and benchmark, comprising 520K frames, 4,549 action clips, and 1.4M 2D/3D point trajectories from diverse robotic and manual procedures under unified action categories. To make annotation scalable, we design a unified framework for surgical 3D point tracking that converts uncalibrated monocular surgical videos into structured 4D annotations through contact-region prompting, active-tool segmentation, dynamic bundle adjustment, and global 4D refinement. This pipeline adapts foundation priors to surgical videos and produces temporally consistent trajectories, pseudo-scale geometry, camera poses, and instrument masks. The dataset captures realistic challenges inherent to surgical environments, including occlusions, smoke, bleeding, and rapid instrument motion. We establish a comprehensive benchmark on SurgPose, SurgT, and STIR, covering both instrument and tissue tracking. We demonstrate instrument displacement estimation, deformable surface reconstruction, and foundation-model fine-tuning. Extensive experiments and analysis demonstrate the effectiveness of our dataset and annotation framework for 3D surgical point tracking, reconstruction, and surgical action understanding. Homepage: https://wrld.github.io/surgtap-3d/
