Latent Dynamics · 2026-10-05
VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning
Mingyu Park, Samyeul Noh, Hyun Myung, Donghwan Lee
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
VIGOR targets zero-shot visual generalization in latent-space model-based RL by fixing a “two-level vulnerability”: visual distractions first perturb encoder latents, and the resulting errors compound through recursive latent rollouts. The authors propose AW (asymmetric weak-to-strong augmentation), DC (dynamics-level consistency via latent regression), and ES (encoder-level stabilization) on top of TD-MPC2.
Why it matters
The authors argue that robustness methods from model-free RL (single-step encoder robustness) do not address recursive rollout compounding in model-based latent dynamics. VIGOR aims to preserve TD-MPC2’s sample efficiency while improving zero-shot robustness to unseen visual distractions, evaluated under the RL-ViGen protocol on both DMC and Robosuite.
Method
- Asymmetric Weak-to-Strong Augmentation (AW): split each batch into weak-only and weak-to-strong subsets and encode paired latents within a single mini-batch.
- Dynamics-Level Consistency (DC): enforce augmentation-invariant transition predictions by aligning recursive latent rollouts from mixed inputs to weakly-augmented successor latents via direct latent regression.
- Encoder-Level Stabilization (ES): add an auxiliary loss aligning weak-only and weak-to-strong encodings to prevent encoder drift induced by DC’s cross-augmentation supervision.
Limitation
Zero-shot generalization under geometric viewpoint rotations remains an open challenge: “Performance drops sharply for all methods, indicating that zero-shot generalization under geometric viewpoint changes remains an open challenge.”
Abstract (from arXiv)
Model-based reinforcement learning (MBRL) achieves strong sample efficiency by planning within learned latent dynamics, yet its performance degrades substantially under unseen visual distractions such as background variations, lighting changes, or camera shifts. Unlike model-free RL, where encoder perturbations affect only single-step predictions, MBRL suffers from a two-level vulnerability: visual distractions first push encoder outputs out of distribution, and these errors then compound through recursive latent rollouts over the planning horizon. We propose visual generalization via latent-space consistency in model-based RL (VIGOR), a framework that enables zero-shot generalization to unseen visual distractions while retaining the sample efficiency of its MBRL backbone. VIGOR integrates three interdependent components: (i) asymmetric weak-to-strong augmentation, which pairs weak-only and weak-to-strong latent views within a single batch; (ii) dynamics-level consistency, which enforces augmentation-invariant transition predictions through direct latent regression; and (iii) encoder-level stabilization, which prevents encoder drift under the cross-augmentation supervision imposed by dynamics-level consistency. Evaluations on the DeepMind Control Suite (DMC) and Robosuite show that VIGOR outperforms state-of-the-art model-free and model-based baselines, surpassing the second-best baseline by 3.4% on DMC and 43.6% on Robosuite. Ablations further show that VIGOR's robustness is augmentation-agnostic: replacing the default augmentation with alternatives from distinct perturbation families preserves strong generalization, confirming that latent-space consistency, not the augmentation choice, drives robustness.
Related papers
- DRILL: Training World Models to Improve Policies, Not Predict Pixels
- Sparse-TD-JEPA: Improving Zero-Shot Reinforcement Learning With Sparse Latent Task Representations
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
- Beyond One-Step Accuracy: State-Affine Latent Transition for Reliable Visual Planning
- Robot Generalization Beyond Volume: The Diversity Path via Zero-Shot Capabilities
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC
VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.02801
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Model-based reinforcement learning (MBRL) achieves strong sample efficiency by planning within learned latent dynamics, yet its performance degrades substantially under unseen visual distractions such as background variations, lighting changes, or camera shifts. Unlike model-free RL, where encoder perturbations affect only single-step predictions, MBRL suffers from a two-level vulnerability: visual distractions first push encoder outputs out of distribution, and these errors then compound through recursive latent rollouts over the planning horizon. We propose visual generalization via latent-space consistency in model-based RL (VIGOR), a framework that enables zero-shot generalization to unseen visual distractions while retaining the sample efficiency of its MBRL backbone. VIGOR integrates three interdependent components: (i) asymmetric weak-to-strong augmentation, which pairs weak-only and weak-to-strong latent views within a single batch; (ii) dynamics-level consistency, which enforces augmentation-invariant transition predictions through direct latent regression; and (iii) encoder-level stabilization, which prevents encoder drift under the cross-augmentation supervision imposed by dynamics-level consistency. Evaluations on the DeepMind Control Suite (DMC) and Robosuite show that VIGOR outperforms state-of-the-art model-free and model-based baselines, surpassing the second-best baseline by 3.4% on DMC and 43.6% on Robosuite. Ablations further show that VIGOR's robustness is augmentation-agnostic: replacing the default augmentation with alternatives from distinct perturbation families preserves strong generalization, confirming that latent-space consistency, not the augmentation choice, drives robustness.