Driving · 2026-10-06
Optimal Control with Learned Critics under Unmodeled State Dependencies
Philipp Schoch, Markus Ryll
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors present a learning-based MPC that augments a nominal analytical model with a residual dynamics network to handle unmodeled state-dependent effects, and blends a learned action-value critic into a constrained iLQR planner to reduce finite-horizon myopia. A GPU-accelerated batched ALM-iLQR solver evaluates learned components inside the control loop and solves thousands of trajectory optimizations in parallel in simulation.
Why it matters
The work targets MPC’s limitations when relevant state dependencies are unmodeled (e.g., contact-like discontinuities) and when finite-horizon planning is insufficient. By integrating learned residual dynamics and a learned critic into constrained iLQR/MPC (rather than using a fully model-free policy), the authors aim to preserve constraint-aware, structured trajectory optimization while improving robustness under dynamics mismatch.
Method
- Blend MPC’s finite-horizon Q-estimate with a learned critic via a recursive blended-Q objective; the blend weight can be applied across all stages or only at the terminal stage.
- Learn residual dynamics by masking state dimensions whose missing evolution is corrected by a neural residual; train with one-step finite-difference targets plus rollout-consistency losses on contiguous fragments.
- Make the approach practical with a GPU-accelerated batched ALM-iLQR solver: learned dynamics and critic networks are evaluated inside the optimal-control loop; constraints handled via an augmented Lagrangian method (ALM).
Limitation
The learned critic degrades at long horizons because it supplies imperfect gradients and curvature at every stage; additionally, the weight of the nominal rollout decays as (γλ)^k, and both effects can destabilize the backward pass over long horizons.
Abstract (from arXiv)
Model Predictive Control (MPC) provides a structured and constraint-aware mechanism for decision-making, but its reliance on optimization-friendly analytical dynamics models limits its use in tasks with contacts and other hard-to-model state dependencies. Model-free reinforcement learning avoids explicit modeling assumptions but typically requires large amounts of interaction data. We present a learning-based MPC framework that combines the data efficiency and structure of local model-based planning with learned components that compensate for incomplete dynamics and finite-horizon myopia. The method augments a nominal analytical model with a residual dynamics network that learns missing state-dependent effects from data and combines the resulting planner with a learned action-value critic that injects long-horizon MDP structure into the local iLQR optimization. To make this practical at reinforcement-learning scale, we develop a GPU-accelerated batched iLQR solver that evaluates learned dynamics and critic networks inside the optimal-control loop and solves thousands of trajectory-optimization problems in parallel. The complete system is integrated into a robotics simulator, enabling scalable model-based reinforcement learning under incomplete dynamics. Experiments on biased and incompletely modeled control tasks show that the approach improves closed-loop control performance while preserving the model-based structure needed for efficient constrained trajectory optimization.
Related papers
- TD-MPC2: Scalable, Robust World Models for Continuous Control
- Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models
- Think Fast, Plan Selectively: Adaptive Deliberation for Efficient Data-Driven MPC
- A Decade of Learning-Based Control: From Theoretic Foundations to Practical Deployment
- Higher-Order Action Supervision Makes A Strong Policy Class
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-04 · Indexed by Pub-AI: 2026-10-06 UTC
Optimal Control with Learned Critics under Unmodeled State Dependencies
Preprint posted: 2026-10-04 · Original record ↗
- arxiv
- 2610.05359
Explicit source versions (1)
- v1 · 2026-10-04
- Preprint available
Source date: 2026-10-04 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Model Predictive Control (MPC) provides a structured and constraint-aware mechanism for decision-making, but its reliance on optimization-friendly analytical dynamics models limits its use in tasks with contacts and other hard-to-model state dependencies. Model-free reinforcement learning avoids explicit modeling assumptions but typically requires large amounts of interaction data. We present a learning-based MPC framework that combines the data efficiency and structure of local model-based planning with learned components that compensate for incomplete dynamics and finite-horizon myopia. The method augments a nominal analytical model with a residual dynamics network that learns missing state-dependent effects from data and combines the resulting planner with a learned action-value critic that injects long-horizon MDP structure into the local iLQR optimization. To make this practical at reinforcement-learning scale, we develop a GPU-accelerated batched iLQR solver that evaluates learned dynamics and critic networks inside the optimal-control loop and solves thousands of trajectory-optimization problems in parallel. The complete system is integrated into a robotics simulator, enabling scalable model-based reinforcement learning under incomplete dynamics. Experiments on biased and incompletely modeled control tasks show that the approach improves closed-loop control performance while preserving the model-based structure needed for efficient constrained trajectory optimization.