Latent Dynamics · 2026-10-30

Generating Critic-Guided Synthetic Experience in Latent World Models

Jorge de Freitas, Christos Ziakas, Alessandra Russo

arXiv:2610.39100PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose Critic-Guided Synthetic Experience (CGSE), which uses a frozen latent world model as a local transition generator for offline RL. Starting from latent states in an offline dataset, CGSE samples candidate actions, refines them to maximize a first-order critic approximation within a trust region, rolls out one step in the frozen world model, and appends the best synthetic transition to the replay buffer. No extra real-environment interaction or test-time planner is used.

Why it matters

The paper targets a common limitation of world-model RL in latent space: compounding errors during imagined rollouts. CGSE avoids long-horizon value backup by storing only one-step synthetic transitions and keeping the offline critic frozen, using critic-guided action refinement to generate synthetic experience offline. The authors report improvements on OGBench manipulation tasks without additional real interaction or test-time planning.

Method

  • CGSE augments an offline dataset by sampling candidate actions from a fixed offline policy and refining each with Flow Q-learning/Q-guidance (FMQ) to maximize a first-order critic approximation within a trust region.
  • A frozen latent action-conditioned world model is used to generate one-step synthetic successors for each refined action; the best action is chosen by the (frozen) critic and the synthetic transition is appended to replay.
  • Actor-only offline policy improvement continues with mixed batches from a replay buffer initialized with the offline dataset, while the offline critic and world model remain frozen.

Limitation

The authors state that “The generator is useful only in a local operating regime,” and that “weaker learned dynamics on scene and puzzle-3x3 (§ 4.3) likely limit the benefit of model-guided generation on these tasks.”

Abstract (from arXiv)

Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world. World models offer an alternative way to generate new experience through a learned dynamics model, yet learning policies in latent space is limited due to compounding errors in imagined rollouts. To this end, we propose using the world model as a local transition generator, leveraging critic guidance to choose actions inside that model to generate synthetic experiences. We study action refinement methodologies, introducing Critic-Guided Synthetic Experience (CGSE) generation in world models. Specifically, CGSE augments an existing offline dataset by sampling candidate actions using an offline policy and refining each to maximize a first-order approximation of the $Q$-function within a trust region. We then use a frozen latent world model to imagine the future state given each updated action, and use the critic to choose the best one, appending this synthetic transition to the replay buffer. Our method requires no additional real-environment interaction or test-time planner, as it generates synthetic experience offline using the world model. On three OGBench manipulation tasks, CGSE improves an offline policy by a relative improvement of $7.1\%$ on the average success rate.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC

preprint

Generating Critic-Guided Synthetic Experience in Latent World Models

Jorge de Freitas, Christos Ziakas, Alessandra Russo

Preprint posted: 2026-10-29 · Original record ↗

arxiv
2610.39100
Explicit source versions (1)
  • v1 · 2026-10-29
  1. Preprint available

    Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world. World models offer an alternative way to generate new experience through a learned dynamics model, yet learning policies in latent space is limited due to compounding errors in imagined rollouts. To this end, we propose using the world model as a local transition generator, leveraging critic guidance to choose actions inside that model to generate synthetic experiences. We study action refinement methodologies, introducing Critic-Guided Synthetic Experience (CGSE) generation in world models. Specifically, CGSE augments an existing offline dataset by sampling candidate actions using an offline policy and refining each to maximize a first-order approximation of the $Q$-function within a trust region. We then use a frozen latent world model to imagine the future state given each updated action, and use the critic to choose the best one, appending this synthetic transition to the replay buffer. Our method requires no additional real-environment interaction or test-time planner, as it generates synthetic experience offline using the world model. On three OGBench manipulation tasks, CGSE improves an offline policy by a relative improvement of $7.1\%$ on the average success rate.