Driving · 2026-10-15
Back on Track: Deployment-time Policy Improvement via World Model Planning
Colton Crosby, Hao Wang, Mac Schwager, Somil Bansal
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose an offline-trained monitoring-and-intervention wrapper for black-box visuomotor policies. They learn a policy-conditioned liveness value to monitor progress during deployment and train a compact latent world model from the same rollouts plus play data. When the monitor detects loss of progress, a short-horizon planner searches the world model for corrective action chunks using the same value as objective, then returns control to the nominal policy.
Why it matters
Many strong visuomotor policies can fail during deployment and cannot recover on their own. The paper targets deployment-time improvement without modifying or retraining the nominal policy and assumes only black-box query access. The results on LIBERO-Goal and a Franka Panda hardware task report improved success while maintaining real-time control via compact latent planning and parallel evaluation.
Method
- Learn a policy-conditioned liveness value from a limited set of labeled on-policy rollouts; the monitor triggers when the smoothed ensemble value crosses a time-varying threshold.
- Train a compact latent world model over a state representation (image encoder features + proprioceptive state) using rollout-length curriculum and auxiliary play data.
- At trigger time, use a short-horizon planner (CEM) to search the world model for corrective action sequences, scoring imagined futures by the same liveness value, then execute an action chunk and return to the nominal policy.
Limitation
The method does not guarantee that every intervention improves the execution: an inaccurate liveness value can trigger when the nominal policy would have completed, and errors in the learned world model can compound over autoregressive rollouts and be exploited by the planner. The framework also requires labeled rollouts of the nominal policy and auxiliary transition data, so it is not a zero-shot improvement method.
Abstract (from arXiv)
State-of-the-art visuomotor policies are increasingly capable of executing complex tasks, yet can fail during deployment in ways they cannot predict or recover from on their own. Improving their performance and robustness typically requires collecting more data, retraining, or modifying the policy architecture, all of which are time- and resource-intensive in practice. We instead propose an offline-trained monitoring-and-intervention framework that improves a policy during deployment using only black-box query access. From a limited set of labeled on-policy rollouts, we learn a policy-conditioned liveness value that measures the nominal policy's progress toward task completion; we additionally train a compact latent world model using these rollouts together with play data. At deployment, the liveness value continuously monitors execution. When it detects a loss of progress, a short-horizon planner searches the world model for corrective actions using the same value as its planning objective, after which control returns to the nominal policy. We evaluate our method on the LIBERO-Goal task suite and on a Franka Panda manipulator, using the vision-language-action model $\pi_{0.5}$ and a diffusion policy as nominal policies, and demonstrate significant improvements in task success rates, while supporting real-time control. Additional results and videos are available on the project website (https://policy-improvement-wm.pages.dev/).
Related papers
- Don't Throw Away the Tail: Action Upcycling for Policy Acceleration
- How Much Planning Is Enough? Reducing Search and Computation in World-Model Planning
- Plans Are Not Executions: Adapting Frozen World-Model Planners to Controller-Induced Execution Shift
- RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations
- Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
Back on Track: Deployment-time Policy Improvement via World Model Planning
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.19579
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
State-of-the-art visuomotor policies are increasingly capable of executing complex tasks, yet can fail during deployment in ways they cannot predict or recover from on their own. Improving their performance and robustness typically requires collecting more data, retraining, or modifying the policy architecture, all of which are time- and resource-intensive in practice. We instead propose an offline-trained monitoring-and-intervention framework that improves a policy during deployment using only black-box query access. From a limited set of labeled on-policy rollouts, we learn a policy-conditioned liveness value that measures the nominal policy's progress toward task completion; we additionally train a compact latent world model using these rollouts together with play data. At deployment, the liveness value continuously monitors execution. When it detects a loss of progress, a short-horizon planner searches the world model for corrective actions using the same value as its planning objective, after which control returns to the nominal policy. We evaluate our method on the LIBERO-Goal task suite and on a Franka Panda manipulator, using the vision-language-action model $π_{0.5}$ and a diffusion policy as nominal policies, and demonstrate significant improvements in task success rates, while supporting real-time control. Additional results and videos are available on the project website (https://policy-improvement-wm.pages.dev/).