Driving · 2026-11-11

Audited Gradients, Anchored Policies: Model-Based Reinforcement Learning for Efficient Flow Control

Ruige Kong, Stefania Fresca

arXiv:2611.13203PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors address a failure mode in pathwise model-based RL for flow control: a differentiable surrogate can track flow trajectories yet misestimate the action gradient used for policy optimization. They audit surrogate action gradients against high-fidelity central finite differences on held-out states, then fit an extracted low-dimensional feedback law from audited gradients and initialize/anchor a neural policy. Tested on four flow cases, their controllers outperform or match a model-free baseline using 3–156× fewer high-fidelity solver interactions.

Why it matters

Action-gradient errors can break policy optimization even when prediction accuracy is high. The paper proposes an explicit pre-optimization check (an “audit”) of surrogate action gradients against high-fidelity finite differences, and uses the audit outcome to decide whether to rely on surrogate gradients or measured solver gradients when constructing a controller. It also studies where audit pass/fail is local (depends on action/state), which matters for reliable deployment in differentiable model-based control.

Method

  • Train differentiable surrogates, compute surrogate action gradients by automatic differentiation, and compute independent high-fidelity central finite-difference gradients on held-out states under fixed acceptance criteria (pooled correlation, sign agreement, and median actuator correlation).
  • If the surrogate passes the audit, fit a low-dimensional feedback law to the surrogate action gradient; otherwise fit to the solver finite-difference gradient (or refit using a different surrogate that passes). Project this law into a neural policy and use an anchor penalty during pathwise optimization.
  • Evaluate each final controller with the high-fidelity solver; include the finite-difference computation used for auditing in the model-based simulation cost, and compare against model-free baselines trained under the same protocol.

Limitation

The paper reports that an audit pass can be local: “a surrogate that passes the audit at one fixed action fails it around the actions the deployed controller applies,” so gradient must be measured where the controller operates.

Abstract (from arXiv)

Feedback controllers for flows are expensive to design, since each candidate must be evaluated with a high-fidelity solver. Differentiable surrogates reduce that cost, but they also introduce a failure mode that prediction error cannot detect. Policy optimization uses the derivative of the objective with respect to the action, which we call the action gradient and which can be computed either through the surrogate or through the high-fidelity solver. Optimization through the surrogate uses only its action gradient, and a surrogate can follow the flow trajectory closely while misestimating this derivative. The surrogate action gradient is compared against a central finite-difference estimate from the high-fidelity solver on held-out states, under acceptance criteria fixed before the comparison, and we call that comparison an audit. A low-dimensional feedback law is then fitted to the surrogate action gradient when the surrogate passes the audit, and to the solver estimate when it does not. This law initializes a neural policy, and a quadratic penalty holds the policy near it during optimization. The proposed approach is validated on four test cases. The resulting controllers outperform or match a model-free baseline, while using 3 to 156 times fewer high-fidelity solver interactions across the four test cases. One surrogate predicts the objective at correlation 0.9996 but its action gradient at only 0.459, so prediction accuracy does not identify a usable surrogate. Another surrogate, whose action gradient agrees with the solver estimate at correlation 0.996, makes the objective 0.52% worse when the policy is optimized through it without the anchor penalty, so an accurate gradient alone does not provide a working controller. Finally, a surrogate that passes the audit at one fixed action fails it around the actions the deployed controller applies, so a pass is local.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-11 · Indexed by Pub-AI: 2026-11-11 UTC

preprint

Audited Gradients, Anchored Policies: Model-Based Reinforcement Learning for Efficient Flow Control

Ruige Kong, Stefania Fresca

First announced by arXiv: 2026-11-11 · Original record ↗

arxiv
2611.13203
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-11 · First detected: 2026-11-11 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.13203v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Feedback controllers for flows are expensive to design, since each candidate must be evaluated with a high-fidelity solver. Differentiable surrogates reduce that cost, but they also introduce a failure mode that prediction error cannot detect. Policy optimization uses the derivative of the objective with respect to the action, which we call the action gradient and which can be computed either through the surrogate or through the high-fidelity solver. Optimization through the surrogate uses only its action gradient, and a surrogate can follow the flow trajectory closely while misestimating this derivative. The surrogate action gradient is compared against a central finite-difference estimate from the high-fidelity solver on held-out states, under acceptance criteria fixed before the comparison, and we call that comparison an audit. A low-dimensional feedback law is then fitted to the surrogate action gradient when the surrogate passes the audit, and to the solver estimate when it does not. This law initializes a neural policy, and a quadratic penalty holds the policy near it during optimization. The proposed approach is validated on four test cases. The resulting controllers outperform or match a model-free baseline, while using 3 to 156 times fewer high-fidelity solver interactions across the four test cases. One surrogate predicts the objective at correlation 0.9996 but its action gradient at only 0.459, so prediction accuracy does not identify a usable surrogate. Another surrogate, whose action gradient agrees with the solver estimate at correlation 0.996, makes the objective 0.52% worse when the policy is optimized through it without the anchor penalty, so an accurate gradient alone does not provide a working controller. Finally, a surrogate that passes the audit at one fixed action fails it around the actions the deployed controller applies, so a pass is local.