Video World Models · 2026-10-21
EndoWorld-JEPA: Motion-Guided Global-Local Latent World Modeling for Endoscopic Video Representation Learning
Jie Wang, Shilong Yang, Tao Zhou, Juan Yu, Rong Qu, Jianfeng Ren, Zhicheng Zhang, Linlin Shen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
EndoWorld-JEPA is a causal motion-guided latent world-model pre-training method for passive endoscopic video. It uses an Optical Motion-guided (OM) module to decompose dense flow into a temporal motion condition and a patch-wise local-motion magnitude map, then predicts next latent states with complementary global and motion-local JEPA objectives. A common pre-trained checkpoint transfers strongly across four downstream endoscopy benchmarks.
Why it matters
The authors target a key gap in passive endoscopy pre-training: observed dynamics from camera motion, tissue deformation, and instruments are often treated implicitly or require recorded controls/poses. EndoWorld-JEPA instead conditions latent next-state prediction on motion observed up to the current frame, aiming to learn predictable visual–motion dynamics for transferable endoscopic representations without reconstructing stochastic appearance artifacts.
Method
- Reformulate passive endoscopic pre-training as causal motion-guided latent-state prediction: given current global and motion-local latent states plus a standardized causal motion window, predict next latent targets in representation space.
- Optical Motion-guided (OM) module: compute dense optical flow, perform dominant–residual decomposition, build an ordered temporal motion descriptor (with displacement, dispersion, and residual statistics) and a residual local-motion magnitude map; convert the map into patch-wise motion weights.
- Use complementary global and motion-local JEPA predictors: temporal condition modulates latent transitions, while local weights define an auxiliary motion-local state; apply patch-wise mixed pooling and past-motion-window dropout to reduce reliance on unreliable motion.
Limitation
The authors state that Farneback flow degrades under rapid motion, large occlusion, weak texture, smoke, and severe illumination change; component-wise median subtraction cannot fully factor rotation, zoom, depth-dependent parallax, or multiple independently moving regions. They also note the fixed K-transition condition is one observation scale and that long-horizon stability, uncertainty calibration, counterfactual control, and planning are outside this one-step formulation; control-aware forecasting would require physical control signals.
Abstract (from arXiv)
Endoscopic video contains state transitions induced by camera displacement, deformable tissue, and instrument interaction, yet existing pre-training methods treat dynamics implicitly. Inspired by action-conditioned joint-embedding predictive world models and by the observation that endoscopic optical flow contains both shared image-plane displacement and localized residual motion, we reformulate passive endoscopic video pre-training as causal motion-guided latent-state prediction. We present EndoWorld-JEPA, to our knowledge the first latent world-model pre-training framework designed for general-purpose endoscopic video representation learning. In place of recorded actions, calibrated poses, or robot kinematics, the model conditions prediction of the next latent state on motion observed up to the current state. Its Optical Motion-guided module separates dominant displacement from residual motion and converts the resulting evidence into an ordered temporal motion condition and spatial local-motion weights. Complementary global and motion-local predictors use these outputs: the temporal condition modulates both latent transitions, whereas the local weights define an auxiliary motion-local state that extends prediction from scene-level evolution to spatially selective dynamics. Patch-wise motion weighting emphasizes localized change while retaining anatomical context, and past-motion-window dropout randomly withholds the complete motion condition to reduce dependence on unreliable motion. This global--local formulation learns predictable visual--motion dynamics in representation space without reconstructing stochastic appearance artifacts. Evaluation shows that a common pre-trained checkpoint achieves strong transfer across four downstream benchmarks, supporting motion-guided global--local latent prediction as an effective pre-training paradigm for endoscopic video.
Related papers
- SonoWorld: Towards Interactive Pose-Controllable Ultrasound World Models
- WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
- Scaling Action-Free Video Latent Prediction for Transferable Representations in End-to-End Autonomous Driving
- RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC
EndoWorld-JEPA: Motion-Guided Global-Local Latent World Modeling for Endoscopic Video Representation Learning
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.27890
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · OAI arXiv created; not header datestamp (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Endoscopic video contains state transitions induced by camera displacement, deformable tissue, and instrument interaction, yet existing pre-training methods treat dynamics implicitly. Inspired by action-conditioned joint-embedding predictive world models and by the observation that endoscopic optical flow contains both shared image-plane displacement and localized residual motion, we reformulate passive endoscopic video pre-training as causal motion-guided latent-state prediction. We present EndoWorld-JEPA, to our knowledge the first latent world-model pre-training framework designed for general-purpose endoscopic video representation learning. In place of recorded actions, calibrated poses, or robot kinematics, the model conditions prediction of the next latent state on motion observed up to the current state. Its Optical Motion-guided module separates dominant displacement from residual motion and converts the resulting evidence into an ordered temporal motion condition and spatial local-motion weights. Complementary global and motion-local predictors use these outputs: the temporal condition modulates both latent transitions, whereas the local weights define an auxiliary motion-local state that extends prediction from scene-level evolution to spatially selective dynamics. Patch-wise motion weighting emphasizes localized change while retaining anatomical context, and past-motion-window dropout randomly withholds the complete motion condition to reduce dependence on unreliable motion. This global--local formulation learns predictable visual--motion dynamics in representation space without reconstructing stochastic appearance artifacts. Evaluation shows that a common pre-trained checkpoint achieves strong transfer across four downstream benchmarks, supporting motion-guided global--local latent prediction as an effective pre-training paradigm for endoscopic video.