Latent Dynamics · 2026-11-11

Few-Shot Residual Adaptation of Pretrained Latent World Models

Jakub Suliga, Pankhuri Vanjani, Nisarga Nilavadi, Wolfram Burgard, Rudolf Lioutikov

arXiv:2611.13592PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors study few-shot residual adaptation for pretrained latent world models under physical dynamics shifts. They keep the pretrained DUET-DINO/DINOv3 components frozen and learn self-supervised residual corrections from latent prediction errors, applied both inside the latent predictor and to its predicted latent state. On a real robot with center-of-mass and friction shifts, frozen success drops near zero; with three adaptation episodes, dual-site recovers 90% (CoM) and 100% (friction).

Why it matters

Pretrained latent world models can fail when deployed under changed physical dynamics. The paper shows that adaptation can be done from only a few self-supervised interaction episodes without modifying pretrained weights or identifying the changed physical parameters, using lightweight residual corrections placed at two sites in the latent dynamics pipeline.

Method

  • Freeze the pretrained latent world model; learn lightweight residual adapters self-supervised from latent prediction errors between predicted and observed transitions (no rewards or dynamics-parameter knowledge).
  • Use dual-site corrections: Adapter+ modules inserted in every predictor block (after attention residual and after block completion) and a token-wise residual MLP added on the predicted latent state; both residuals are zero-initialized.
  • Alternate model-predictive control (CEM) with online adaptation over three interaction episodes; adapters are trained self-supervised using cached encoder latents and a one-step objective.

Limitation

The authors’ current study is limited to center-of-mass and friction shifts in a pushing task and learns a separate residual correction for each dynamics setting; it may also lose performance when returned to general dynamics.

Abstract (from arXiv)

Pretrained latent world models can support visual planning and control, but their predictions may fail when physical dynamics change at deployment. We study whether such models can adapt from only a few interactions without modifying their pretrained weights or identifying the changed physical parameters. We introduce a few-shot residual adaptation approach that learns self-supervised corrections from latent prediction errors. The corrections are applied both inside the latent predictor and directly to its predicted latent state, while the pretrained world model remains frozen. We evaluate the approach in simulation and on a real robot under center-of-mass and friction shifts. These changes reduce the success of the frozen model to near zero. Using three interaction episodes for adaptation, our dual-site approach recovers 90\% success under the center-of-mass shift and 100\% under the modified friction on the real robot. Ablations show that correcting both locations outperforms single-site adaptation. Our diagnostics further suggest that much of the learned correction reduces a general prediction mismatch, while successful control also depends on shift-specific information captured during adaptation.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-10 · Indexed by Pub-AI: 2026-11-11 UTC

preprint

Few-Shot Residual Adaptation of Pretrained Latent World Models

Jakub Suliga, Pankhuri Vanjani, Nisarga Nilavadi, Wolfram Burgard, Rudolf Lioutikov

Preprint posted: 2026-11-10 · Original record ↗

arxiv
2611.13592
Explicit source versions (1)
  • v1 · 2026-11-10
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-11 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Pretrained latent world models can support visual planning and control, but their predictions may fail when physical dynamics change at deployment. We study whether such models can adapt from only a few interactions without modifying their pretrained weights or identifying the changed physical parameters. We introduce a few-shot residual adaptation approach that learns self-supervised corrections from latent prediction errors. The corrections are applied both inside the latent predictor and directly to its predicted latent state, while the pretrained world model remains frozen. We evaluate the approach in simulation and on a real robot under center-of-mass and friction shifts. These changes reduce the success of the frozen model to near zero. Using three interaction episodes for adaptation, our dual-site approach recovers 90\% success under the center-of-mass shift and 100\% under the modified friction on the real robot. Ablations show that correcting both locations outperforms single-site adaptation. Our diagnostics further suggest that much of the learned correction reduces a general prediction mismatch, while successful control also depends on shift-specific information captured during adaptation.