Robotics · 2026-11-11

Koopman Model-Based Reinforcement Learning

Wenjian Hao, Paulo C. Heredia, Shaoshuai Mou

arXiv:2611.14186PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose Koopman model-based reinforcement learning (K-MBRL) for controlling nonlinear robotic systems with unknown dynamics. K-MBRL learns a stochastic deep Koopman model for dynamics in a linear lifted representation and a critic for future cost. It approximates the policy objective using immediate cost plus one-step predicted-state value, aiming to reduce rollout prediction error accumulation and enable real-time feedforward control without online optimization.

Why it matters

The paper targets a practical difficulty in model-based RL: how to obtain computationally efficient policy improvement while limiting prediction-error accumulation from long model rollouts. By combining one-step Koopman dynamics propagation with critic-based future cost estimation, the authors aim for efficient policy gradients (via linear structure) and real-time control using feedforward evaluation, plus convergence analysis under stochastic approximation assumptions.

Method

  • Learn unknown nonlinear dynamics using a stochastic deep Koopman model (DKM) that approximates the dynamics in a linear representation in a lifted space.
  • Approximate the policy objective by combining analytical immediate cost with a critic-estimated future cost evaluated at the one-step predicted state.
  • Perform iterative policy optimization using this one-step propagation scheme to mitigate prediction error accumulation and achieve real-time feedforward control without online optimization.
Abstract (from arXiv)

We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-10 · Indexed by Pub-AI: 2026-11-11 UTC

preprint

Koopman Model-Based Reinforcement Learning

Wenjian Hao, Paulo C. Heredia, Shaoshuai Mou

Preprint posted: 2026-11-10 · Original record ↗

arxiv
2611.14186
Explicit source versions (1)
  • v1 · 2026-11-10
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-11 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.