Driving · 2026-10-14

Learning to See through Action: Predictive Curiosity for Robots

Stelios Zarifis, Panagiotis Filntisis, George Retsinas, Petros Maragos

arXiv:2610.14302PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors introduce DySCOVER, a self-supervised active visual learning framework for robots. A robot uses a visual autoencoder to encode RGB observations into latents and trains an action-conditioned recurrent world model to predict the next visual latent. The policy is trained with intrinsic reward that mixes visual reconstruction error (visual novelty) and world-model prediction error (predictive curiosity), improving visual reconstruction over visual-curiosity-only across multiple settings.

Why it matters

The paper tests whether robots can improve visual representation learning by actively choosing actions based on how predictable the visual consequences are. The authors report that adding predictive curiosity (world-model prediction error) consistently improves visual reconstruction compared to reconstruction-based curiosity alone, including with compact replay buffers and across robot embodiments and object-augmented scenes.

Method

  • DySCOVER trains a visual autoencoder on RGB frames and a recurrent action-conditioned world model to predict the next visual latent state given current latent and action.
  • The policy is trained only with intrinsic reward: reconstruction error for visually novel observations, plus world-model prediction error for transitions with consequences that are difficult to predict.
  • A slowly updated reference predictor adjusts the world-model reward by using a log-ratio between current prediction error and a predicted reference error.

Limitation

The paper evaluates primarily via visual reconstruction quality (MSE/PSNR) on independently created, unseen evaluation frames, and notes future work includes evaluation on external tasks to investigate whether the learned representations transfer to complex scenarios such as planning or manipulation.

Abstract (from arXiv)

Humans and animals do not learn vision from a fixed set of data: they move, interact, and learn from the visual consequences of their actions. Motivated by this developmental view, we study whether robots can improve visual learning by actively collecting experience through curiosity-driven interaction. We introduce DySCOVER, a self-supervised framework for active robot visual learning. A robot observes RGB frames, learns a visual autoencoder that encodes and reconstructs them, and trains a recurrent world model to predict the next visual latent state conditioned on the current state and action. The agent's policy is trained only with intrinsic rewards: reconstruction error encourages visually novel observations, while the world model's error encourages transitions whose consequences are not yet predictable. This prediction error biases the policy toward more intricate trajectories that enrich the agent's set of experiences with observations beyond those encouraged by visual novelty alone. Our experiments conducted with robotic arms as the embodied agents show that adding predictive curiosity improves the vision model over using visual curiosity alone. Across variations in initial conditions, experience capacity, embodiment, and scene content, adding predictive curiosity consistently improved visual learning. These results show that predictive curiosity can improve active visual learning beyond reconstruction-based curiosity alone.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-14 UTC

preprint

Learning to See through Action: Predictive Curiosity for Robots

Stelios Zarifis, Panagiotis Filntisis, George Retsinas, Petros Maragos

Preprint posted: 2026-10-09 · Original record ↗

arxiv
2610.14302
Explicit source versions (1)
  • v1 · 2026-10-09
  1. Preprint available

    Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Humans and animals do not learn vision from a fixed set of data: they move, interact, and learn from the visual consequences of their actions. Motivated by this developmental view, we study whether robots can improve visual learning by actively collecting experience through curiosity-driven interaction. We introduce DySCOVER, a self-supervised framework for active robot visual learning. A robot observes RGB frames, learns a visual autoencoder that encodes and reconstructs them, and trains a recurrent world model to predict the next visual latent state conditioned on the current state and action. The agent's policy is trained only with intrinsic rewards: reconstruction error encourages visually novel observations, while the world model's error encourages transitions whose consequences are not yet predictable. This prediction error biases the policy toward more intricate trajectories that enrich the agent's set of experiences with observations beyond those encouraged by visual novelty alone. Our experiments conducted with robotic arms as the embodied agents show that adding predictive curiosity improves the vision model over using visual curiosity alone. Across variations in initial conditions, experience capacity, embodiment, and scene content, adding predictive curiosity consistently improved visual learning. These results show that predictive curiosity can improve active visual learning beyond reconstruction-based curiosity alone.