Latent Dynamics · 2026-10-02

iSEE: Object Permanence Through Self-Supervision

Pramish Paudel, Ajad Chhatkuli, Luc Van Gool, Danda Pani Paudel

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

iSEE is a label-free, self-supervised object-centric video model for object permanence under occlusion. It adds temporal evidence normalization (TEN) to detect occlusion and hold a slot’s appearance, separates appearance and position into two streams, and trains a walker to follow hidden positions using only where the object reappears. On LA-CATER static, iSEE returns reappearing objects to their own slot after 86% of occlusions, and localizes hidden objects within 4.1 mAP of supervised RAM.

Why it matters

The authors address object permanence in self-supervised slot-based video representations, where slots are otherwise lost during occlusion. iSEE combines (1) occlusion detection from slot attention over time (TEN), (2) explicit what/where separation via two slot streams, and (3) a reappearance-trained position walker, enabling tracking, re-identification, localization during hidden periods, and downstream planning using the position stream.

Method

Temporal evidence normalization (TEN): compares a slot’s attention-derived evidence with its own past to detect occlusions and pause the appearance update until reappearance.
Appearance-position separation: splits each slot into appearance and position streams; appearance is held during occlusion while position continues via a walker.
Reappearance-trained walker: follows the hidden object’s position; training uses only the position where the object reappears, with no labels while hidden.

Limitation

Under the moving camera, iSEE scores lower than in the static camera, and the drop “starts upstream of the walker, in the encoder,” which discovers objects less well and returns fewer targets to their slot.

Abstract (from arXiv)

Object permanence, keeping track of an object's identity and position while it is occluded, is central to video representations that track, predict and plan. Trackers that achieve it learn from boxes, track identities and visibility labels. On the other hand, self-supervised object-centric methods discover objects without labels: through slot attention, it represents a video as slots that bind to objects and follow them across frames. However, these slots are lost under occlusion, making the desired permanence impossible. Reasoning permanence is a hard problem because it requires to detect when an object becomes occluded, re-identify when object reappears, and keep the object's hidden position continuous, using reapperance as the only learning cue. To address this, we propose iSEE, a novel framework that offers all three aforementioned requirements, without any labels whatsoever. We built iSEE using the following three proposed components: (i) Object evidence modelling: a slot's attention, compared with its own past, reveals when its object is hidden. (ii) Appearance-position separation: two slot streams let the appearance be held for re-identification while the position keeps changing. (iii) Permanence from reappearance: a walker follows the hidden object's position, trained only on where the object reappears. On LA-CATER static, iSEE returns a reappearing object to its own slot after 86% of occlusions, against 32% for SlotContrast, and localises it while hidden within 4.1 mAP of the label-trained SoTA RAM. The two streams also allow downstream planning, with the position stream as the action of a world model. Project page: https://insait-institute.github.io/iSEE/

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC

preprint