Driving · 2026-10-01

Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models

Kowndinya Boyalakuntla, Yuhan Liu, Abdeslam Boularias

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose PL-MPC, an extension of policy-constrained TD-M(PC)2 that modifies (1) critic supervision via hybrid multi-step TD targets, (2) MPC terminal-value evaluation via disagreement-aware terminal estimates, and (3) actor learning via return-weighted actor distillation from planner-executed actions. They report largest gains on HumanoidBench balance-hard and hurdle, and higher zero-shot sim-to-real success on wrench–nut alignment.

Why it matters

PL-MPC targets the planning–learning loop in learned world-model control: the planner generates experience for learning, while the learned critic and actor affect how the planner scores and selects trajectories. The method specifically changes three interfaces (TD targets, terminal-value scoring, and planner-to-policy transfer) without changing the latent world model or MPPI optimizer, and evaluates both benchmark and real-robot sim-to-real transfer.

Method

Introduce PL-MPC on top of policy-constrained TD-M(PC)2; keep the latent world model and MPPI optimization unchanged.
Use hybrid multi-step TD targets (MTD): observed rewards from planner-generated replay enter earlier critic targets, with learned-model tails only beyond the replay slice.
Modify MPC terminal scoring with disagreement-aware terminal estimates (ATE) and train the actor with return-weighted actor distillation (RAD) from planner-executed actions weighted by realized episode return.

Limitation

The paper reports that performance across the broader benchmark remains task dependent, and that on tasks such as maze useful planner trajectories may be rare and RAD may have little signal to amplify.

Abstract (from arXiv)

Planning with learned world models combines online trajectory optimization with learned value and policy functions for high-dimensional control. Because the planner determines the experience used for learning, while the learned critic and actor in turn score and propose future plans, planning and learning form a closed feedback loop. TD-MPC is a prominent instance of this design. Recent policy-constrained variants strengthen one part of the loop by aligning the learned policy with planner behavior. We introduce PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation. Hybrid multi-step TD targets expose critic updates to more realized rewards before bootstrapping; disagreement-aware terminal estimates reduce the influence of uncertain critic values during MPPI planning; and return-weighted actor distillation emphasizes planner-executed actions from high-return episodes. The world-model architecture and MPPI optimizer are otherwise unchanged. On HumanoidBench, the largest gains occur on \texttt{balance-hard}, where Total Average Return (TAR) increases from $98\pm18$ to $387\pm255$, and \texttt{hurdle}, from $199\pm13$ to $466\pm200$; performance across the broader benchmark remains task dependent, and PL-MPC remains competitive on DMControl. Controlled ablations show different component interactions across the two tasks. We further demonstrate zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2 on the training object size and two unseen sizes. Code and data will be available at: https://pl-mpc-humanoid.github.io.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC

preprint