Latent Dynamics · 2026-10-14

Chunk-wise Model-based Policy Optimization

Weixi Song, Donglin Wang, Yong-Lu Li

arXiv:2610.14564PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

CMPO (Chunk-wise Model-based Policy Optimization) learns a chunk world model and a chunk actor so an H-step hypothetical rollout is generated via a single model invocation. This avoids recurrent model rollout, reduces compounding errors and instability from BPTT, and lowers training/inference cost. The authors evaluate on 19 HumanoidBench tasks and demonstrate real-world sim-to-real transfer for quadruped bipedal walking.

Why it matters

The authors address three bottlenecks in model-gradient MBRL: sequential rollout cost, recurrent compounding errors, and unstable/expensive BPTT. By switching both the policy and dynamics to a chunk-wise interface, CMPO provides a training-time differentiable evaluator without recurrent model invocation, aiming to improve optimization stability and computational efficiency.

Method

  • Uses a chunk world model f(z_t, a_{t:t+H-1}) to predict a latent state chunk z_{t+1:t+H} in one forward pass, together with a chunk actor π(z_t) = a_{t:t+H-1}.
  • Optimizes a multi-step value expansion objective using the chunk world model as a differentiable training-time evaluator, avoiding recurrent model-policy interactions and long-horizon BPTT.
  • At inference, predicts an action chunk but executes only the first action to keep closed-loop responsiveness.

Limitation

In addition, to ensure closed-loop responsiveness in dynamic environments, our chunk policy executes only the first action of each predicted chunk. However, for less dynamic environments, this design implies that the remaining actions within the chunk are discarded and therefore underutilized.

Abstract (from arXiv)

Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: https://song-wx.github.io/cmpo.github.io/

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-10 · Indexed by Pub-AI: 2026-10-14 UTC

preprint

Chunk-wise Model-based Policy Optimization

Weixi Song, Donglin Wang, Yong-Lu Li

Preprint posted: 2026-10-10 · Original record ↗

arxiv
2610.14564
Explicit source versions (1)
  • v1 · 2026-10-10
  1. Preprint available

    Source date: 2026-10-10 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: https://song-wx.github.io/cmpo.github.io/