Driving · 2026-11-13

Fast Inference-Time Policy Refinement via Action-Affine World Models

Run Wang, Yilin Mo, Liang Wu

arXiv:2611.15011PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose an action-affine world model (A²WM) for fast inference-time refinement of a frozen imitation policy. Given interaction history and a policy action proposal, A²WM predicts a nominal trajectory plus a linear response to action corrections over the full horizon. This enables quadratic-program (QP) refinement with one model evaluation and one QP solve per step, without updating policy/model weights. Across four simulated manipulation tasks, it reports the highest success rate among evaluated refinement baselines.

Why it matters

The paper targets a deployment problem: small execution errors can compound, and existing refinement methods may require costly online planning or additional on-policy learning. A²WM is designed so that online correction optimization can be done cheaply (convex QP) by learning local action sensitivities offline and predicting the full multi-step effect directly, avoiding recursive error propagation.

Method

  • Learn an action-affine world model offline from action-perturbed rollouts of a frozen imitation policy; predict horizon geometry as a nominal trajectory plus a linear action-response term.
  • Use direct multi-step prediction (no feeding intermediate predictions back into the model) so the planned horizon is obtained in one model evaluation while corrections are optimized via a convex QP.
  • Online refinement executes only one model evaluation and one QP solve per step; both policy and world-model weights remain frozen during deployment.

Limitation

The current framework assumes measured robot and object states and uses manually specified geometric objectives. Its local predictions also depend on the coverage of the collected rollouts.

Abstract (from arXiv)

Imitation policies can perform complex manipulation, yet small execution errors can accumulate and cause failure. Refinement can correct these errors, but existing approaches often require additional on-policy learning or costly online planning. Our key insight is that a pretrained policy already supplies a useful nominal behavior: refinement can focus on predicting how small action corrections change its outcome. We therefore learn this local response in a form that is cheap to optimize. Our \textit{action-affine world model} (A\textsuperscript{2}WM) learns local action sensitivities offline from action-perturbed policy rollouts. Given interaction history and a policy proposal, it predicts task-relevant geometry as a nominal trajectory plus a linear response to corrections. It predicts the full planning horizon directly, avoiding the propagation of intermediate errors through recursive model evaluations. Quadratic penalties on predicted task errors and linear correction constraints then yield a cheap convex quadratic program (QP). Online refinement requires one model evaluation and one QP solve, while both policy and model weights remain frozen. Across four simulated manipulation tasks, our framework achieves the highest success rate among evaluated baselines, adding only 3.8 ms of mean refinement overhead per control step, excluding base-policy sampling.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

Fast Inference-Time Policy Refinement via Action-Affine World Models

Run Wang, Yilin Mo, Liang Wu

First announced by arXiv: 2026-11-13 · Original record ↗

arxiv
2611.15011
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.15011v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Imitation policies can perform complex manipulation, yet small execution errors can accumulate and cause failure. Refinement can correct these errors, but existing approaches often require additional on-policy learning or costly online planning. Our key insight is that a pretrained policy already supplies a useful nominal behavior: refinement can focus on predicting how small action corrections change its outcome. We therefore learn this local response in a form that is cheap to optimize. Our \textit{action-affine world model} (A\textsuperscript{2}WM) learns local action sensitivities offline from action-perturbed policy rollouts. Given interaction history and a policy proposal, it predicts task-relevant geometry as a nominal trajectory plus a linear response to corrections. It predicts the full planning horizon directly, avoiding the propagation of intermediate errors through recursive model evaluations. Quadratic penalties on predicted task errors and linear correction constraints then yield a cheap convex quadratic program (QP). Online refinement requires one model evaluation and one QP solve, while both policy and model weights remain frozen. Across four simulated manipulation tasks, our framework achieves the highest success rate among evaluated baselines, adding only 3.8 ms of mean refinement overhead per control step, excluding base-policy sampling.