Latent Dynamics · 2026-10-07

Modeling Latent Disturbances for Robust Decision-Making in World Models

Junwon Seo, Andrea Bajcsy

arXiv:2610.07599PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors study robust decision-making in world-model (WM) latent space by introducing a latent-space disturbance that yields pessimistic but plausible WM imaginations. They build an uncertainty set of plausible latent dynamics using a dynamics-aware KL similarity metric plus out-of-distribution detection, calibrate it with conformal prediction, and then jointly optimize robust actions and the worst-case latent disturbances via game-theoretic optimization for robust safety filtering and sample-and-verify steering.

Why it matters

Robust optimization needs a well-defined disturbance set, but WM latent spaces can contain implausible out-of-distribution states where a disturbance could create overly pessimistic (over-conservative) decisions. The authors’ LUCID-style disturbance and conformal calibration aim to constrain worst-case latent imaginations to be plausible, enabling robust policy steering directly in WM latent spaces, including real-robot safety filtering.

Method

  • Model latent-space disturbance as an alternative latent dynamics model perturbing the learned latent dynamics, producing pessimistic but plausible WM imaginations.
  • Construct a plausible-uncertainty set using a dynamics-aware left-KL ball around the learned latent predictive distribution plus out-of-distribution detection to exclude implausible latent states.
  • Calibrate the uncertainty set thresholds with conformal prediction, then jointly optimize robust actions and worst-case latent disturbances via game-theoretic optimization for latent safety filtering and sample-and-verify steering.

Limitation

The work assumes the OOD detection is calibrated so that WM imaginations induced by the latent disturbance remain plausible without becoming overly pessimistic; additionally, their safety-filter training truncates an imagination once the next latent state satisfies s OOD (z t+1 ) > ε OOD.

Abstract (from arXiv)

In this paper, we study robust decision-making in the latent space of world models (WMs). Robust optimization is a mathematical framework where, given explicitly specified dynamics and physically meaningful disturbances, a robot can select actions that remain effective even under worst-case disturbances. However, applying this principle to the learned latent space of WMs introduces a fundamental challenge: because WMs have fully learned state spaces and dynamics inferred from high-dimensional observations, it is unclear how to define latent-space disturbances that faithfully represent uncertainty in the underlying system. Our key idea is to model a latent-space disturbance as a perturbation to the learned latent dynamics that induces pessimistic but plausible transitions. Specifically, we construct a set of plausible latent dynamics by combining a dynamics-aware similarity metric that captures plausible transitions with out-of-distribution detection that excludes implausible latent states. We calibrate this uncertainty set over latent dynamics using conformal prediction, ensuring that WM imaginations induced by the latent disturbance remain plausible without becoming overly pessimistic. We then jointly optimize robust robot actions and the worst-case latent disturbances through game-theoretic optimization. We leverage this latent-space robust optimization to robustify policy steering, considering two paradigms: latent safety filtering and sample-and-verify steering of a generative control policy. Our controlled simulation experiments show that our latent disturbance enables robust decision-making directly in WM latent spaces, and hardware experiments with a Franka manipulator show that modeling latent disturbances enables robust policy steering, reducing failures by 70% in safety filtering and 54% in sampling-based policy steering. Project website: https://junwon.me/LatentDisturbance/.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC

preprint

Modeling Latent Disturbances for Robust Decision-Making in World Models

Junwon Seo, Andrea Bajcsy

Preprint posted: 2026-10-06 · Original record ↗

arxiv
2610.07599
Explicit source versions (1)
  • v1 · 2026-10-06
  1. Preprint available

    Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

In this paper, we study robust decision-making in the latent space of world models (WMs). Robust optimization is a mathematical framework where, given explicitly specified dynamics and physically meaningful disturbances, a robot can select actions that remain effective even under worst-case disturbances. However, applying this principle to the learned latent space of WMs introduces a fundamental challenge: because WMs have fully learned state spaces and dynamics inferred from high-dimensional observations, it is unclear how to define latent-space disturbances that faithfully represent uncertainty in the underlying system. Our key idea is to model a latent-space disturbance as a perturbation to the learned latent dynamics that induces pessimistic but plausible transitions. Specifically, we construct a set of plausible latent dynamics by combining a dynamics-aware similarity metric that captures plausible transitions with out-of-distribution detection that excludes implausible latent states. We calibrate this uncertainty set over latent dynamics using conformal prediction, ensuring that WM imaginations induced by the latent disturbance remain plausible without becoming overly pessimistic. We then jointly optimize robust robot actions and the worst-case latent disturbances through game-theoretic optimization. We leverage this latent-space robust optimization to robustify policy steering, considering two paradigms: latent safety filtering and sample-and-verify steering of a generative control policy. Our controlled simulation experiments show that our latent disturbance enables robust decision-making directly in WM latent spaces, and hardware experiments with a Franka manipulator show that modeling latent disturbances enables robust policy steering, reducing failures by 70% in safety filtering and 54% in sampling-based policy steering. Project website: https://junwon.me/LatentDisturbance/.