Robotics · 2026-11-11
Koopman Model-Based Reinforcement Learning
Wenjian Hao, Paulo C. Heredia, Shaoshuai Mou
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose Koopman model-based reinforcement learning (K-MBRL) for controlling nonlinear robotic systems with unknown dynamics. K-MBRL learns a stochastic deep Koopman model for dynamics in a linear lifted representation and a critic for future cost. It approximates the policy objective using immediate cost plus one-step predicted-state value, aiming to reduce rollout prediction error accumulation and enable real-time feedforward control without online optimization.
Why it matters
The paper targets a practical difficulty in model-based RL: how to obtain computationally efficient policy improvement while limiting prediction-error accumulation from long model rollouts. By combining one-step Koopman dynamics propagation with critic-based future cost estimation, the authors aim for efficient policy gradients (via linear structure) and real-time control using feedforward evaluation, plus convergence analysis under stochastic approximation assumptions.
Method
- Learn unknown nonlinear dynamics using a stochastic deep Koopman model (DKM) that approximates the dynamics in a linear representation in a lifted space.
- Approximate the policy objective by combining analytical immediate cost with a critic-estimated future cost evaluated at the one-step predicted state.
- Perform iterative policy optimization using this one-step propagation scheme to mitigate prediction error accumulation and achieve real-time feedforward control without online optimization.
Abstract (from arXiv)
We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.
Related papers
- Optimal Control with Learned Critics under Unmodeled State Dependencies
- ODE Policy: Robot Learning via Action Dynamics, Not Actions
- DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
- A Decade of Learning-Based Control: From Theoretic Foundations to Practical Deployment
- LocoWM: High-Precision Locomotion through World-Model-Guided Residual Adaptation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-10 · Indexed by Pub-AI: 2026-11-11 UTC
Koopman Model-Based Reinforcement Learning
Preprint posted: 2026-11-10 · Original record ↗
- arxiv
- 2611.14186
Explicit source versions (1)
- v1 · 2026-11-10
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-11 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.14186v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.
