Latent Dynamics · 2026-10-14
Chunk-wise Model-based Policy Optimization
Weixi Song, Donglin Wang, Yong-Lu Li
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
CMPO (Chunk-wise Model-based Policy Optimization) learns a chunk world model and a chunk actor so an H-step hypothetical rollout is generated via a single model invocation. This avoids recurrent model rollout, reduces compounding errors and instability from BPTT, and lowers training/inference cost. The authors evaluate on 19 HumanoidBench tasks and demonstrate real-world sim-to-real transfer for quadruped bipedal walking.
Why it matters
The authors address three bottlenecks in model-gradient MBRL: sequential rollout cost, recurrent compounding errors, and unstable/expensive BPTT. By switching both the policy and dynamics to a chunk-wise interface, CMPO provides a training-time differentiable evaluator without recurrent model invocation, aiming to improve optimization stability and computational efficiency.
Method
- Uses a chunk world model f(z_t, a_{t:t+H-1}) to predict a latent state chunk z_{t+1:t+H} in one forward pass, together with a chunk actor π(z_t) = a_{t:t+H-1}.
- Optimizes a multi-step value expansion objective using the chunk world model as a differentiable training-time evaluator, avoiding recurrent model-policy interactions and long-horizon BPTT.
- At inference, predicts an action chunk but executes only the first action to keep closed-loop responsiveness.
Limitation
In addition, to ensure closed-loop responsiveness in dynamic environments, our chunk policy executes only the first action of each predicted chunk. However, for less dynamic environments, this design implies that the remaining actions within the chunk are discarded and therefore underutilized.
Abstract (from arXiv)
Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: https://song-wx.github.io/cmpo.github.io/
Related papers
- Staircase Policy: Streaming Inference for World-Action Models with Large Action Chunks
- $\nabla$Mimic: Ultra-Efficient Humanoid Control with Differentiable Simulation
- Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models
- ODE Policy: Robot Learning via Action Dynamics, Not Actions
- BeJEPA: A Goal-Conditioned Humanoid Behavior World Model via Unsupervised RL
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-10 · Indexed by Pub-AI: 2026-10-14 UTC
Chunk-wise Model-based Policy Optimization
Preprint posted: 2026-10-10 · Original record ↗
- arxiv
- 2610.14564
Explicit source versions (1)
- v1 · 2026-10-10
- Preprint available
Source date: 2026-10-10 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: https://song-wx.github.io/cmpo.github.io/
