Video World Models · 2026-11-03
TAP-Out: Tracking Any Point in 360{\deg} via Outpainting
Finlay G. C. Hudson, James A. D. Gardner, William A. P. Smith
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TAP-Out is an offline pipeline for tracking points in 360° even after they leave the camera’s field of view. From a narrow-FoV video, it outpaints a complete 360° equirectangular video (Argus), segments/locates the target (SAM2), generates object-centered perspective crops, and runs standard in-frame point tracking on the crops to recover 360° direction tracks.
Why it matters
The authors target out-of-view (OOV) tracking, which requires reasoning about unobserved dynamics. They decouple in-view correspondence and OOV scene completion by using a generative video outpainting model as a world-model prior, enabling explicit completion to improve OOV tracking quality on TAPVid360.
Method
- Outpaint: map perspective frames onto a masked equirectangular canvas using estimated (or ground-truth) camera parameters, then complete the full 360° video with a diffusion-based video-to-360 model (Argus).
- Locate object: run video object segmentation (SAM2) on the completed 360° equirectangular video to obtain a coarse mask and centroid for the tracked object.
- Track in centred view: generate object-centred perspective crops whose optical axis aligns with the tracked object direction, run an in-frame TAP tracker (e.g., CoTracker3) on these crops, then reproject tracks back to 3D directions in the original camera coordinate system.
Limitation
TAP-Out is an offline method, processing the complete video before producing tracks; the authors report total inference time is approximately 470 seconds per video (vs 0.15 seconds for CoTracker360).
Abstract (from arXiv)
Tracking points that leave the camera's field of view (FoV) requires reasoning about unobserved scene dynamics, a fundamentally different problem from tracking visible correspondences. We present TAP-Out, an offline method that makes this out-of-view (OOV) reasoning explicit by leveraging generative video outpainting as a world-model prior. Given a narrow-FoV input video, TAP-Out outpaints a complete 360{\deg} equirectangular video, locates target objects within the panoramic scene via video object segmentation, and synthesises object-centred perspective crops on which standard in-frame trackers operate. This decoupled design composes off-the-shelf foundation models without requiring task-specific architectural modifications. On the TAPVid360 benchmark, TAP-Out achieves state-of-the-art performance, improving OOV tracking accuracy by approximately 3x over the best prior method. We demonstrate its flexibility on out-of-frame object segmentation and detection and we provide detailed diagnostic analyses across motion regimes, static versus dynamic content, and hallucination quality, characterising when and why the approach succeeds or fails.
Related papers
- PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation
- World Observer: Joint Actor-Observer Generation for Persistent World Modeling
- Momentum: Persistent Dynamics for Video World Models
- From Video to Digital Twin: An Exploratory Study of World Models and Gaussian Splatting on In-the-Wild Video
- SPW-Nav: A Streaming Panoramic World Model for Language-Guided Navigation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC
TAP-Out: Tracking Any Point in 360° via Outpainting
First announced by arXiv: 2026-11-03 · Original record ↗
- arxiv
- 2611.00406
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.00406v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Tracking points that leave the camera's field of view (FoV) requires reasoning about unobserved scene dynamics, a fundamentally different problem from tracking visible correspondences. We present TAP-Out, an offline method that makes this out-of-view (OOV) reasoning explicit by leveraging generative video outpainting as a world-model prior. Given a narrow-FoV input video, TAP-Out outpaints a complete 360° equirectangular video, locates target objects within the panoramic scene via video object segmentation, and synthesises object-centred perspective crops on which standard in-frame trackers operate. This decoupled design composes off-the-shelf foundation models without requiring task-specific architectural modifications. On the TAPVid360 benchmark, TAP-Out achieves state-of-the-art performance, improving OOV tracking accuracy by approximately 3x over the best prior method. We demonstrate its flexibility on out-of-frame object segmentation and detection and we provide detailed diagnostic analyses across motion regimes, static versus dynamic content, and hallucination quality, characterising when and why the approach succeeds or fails.
