Driving · 2026-10-05
FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training
Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Daren Zha, Jun Xiao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
FSPO is a feedback-state controller for budgeted LLM RL post-training that jointly handles (i) policy-induced risk shift via a policy-consistent risk-to-go Bellman target using the frozen deployment controller, (ii) selection-induced calibration shift via decision-conditioned trajectory calibration (DCTC) with cross-fitted trajectories, and (iii) multi-resource feasibility via a Pareto resource continuation certificate (PRCC). Under a matched GRPO resource envelope it reports 66.11% held-out and 59.43% OOD accuracy.
Why it matters
Adaptive LLM RL post-training changes multiple training actuators online, creating coupled failures: risk estimates trained from behavior trajectories may not match deployment risk, calibration on logged actions can become miscalibrated after action selection, and independently minimizing each resource may not yield an executable multi-resource continuation. FSPO is designed to align future risk modeling, calibration distribution, and residual resource feasibility with the decisions used at deployment.
Method
- Policy-consistent risk-to-go: learn ensembles for long-horizon utility Q and future stability risk; the Bellman target uses the same frozen controller used for future decisions (policy-conditioned risk-to-go recursion).
- Decision-conditioned trajectory calibration (DCTC): cross-fitted, trajectory-blocked calibration where the provisional controller selects actions; fit calibration on selected decisions and evaluate selected-action calibration separately from all-action calibration.
- Pareto resource continuation certificate (PRCC): compute non-dominated cumulative reservations for token, verifier, and device-time budgets and admit an action only if a realizable residual plan remains; abstain/commit rules are specified by feasibility masks.
Abstract (from arXiv)
Adaptive LLM reinforcement-learning post-training changes multiple training actuators online, including rollout temperature, group size, clipping, KL regularization, verifier allocation, and update budget. Three coupled issues remain unresolved. A future-risk model trained from behavior trajectories need not estimate the risk induced by the controller that will be deployed; a score calibrated on logged state-action pairs can become miscalibrated after selective action choice; and independent per-resource minimum costs do not in general certify a feasible multi-resource continuation. We introduce FSPO, a feedback-state controller for budgeted LLM RL post-training that addresses these issues jointly. FSPO learns a policy-consistent risk-to-go model whose Bellman target follows the same frozen controller used for future decisions, together with a long-horizon utility model. Decision-conditioned trajectory calibration (DCTC) calibrates risk on cross-fitted trajectories generated by actions selected by provisional controllers. A Pareto resource continuation certificate (PRCC) admits an action only when a non-dominated cumulative reservation remains feasible over the residual horizon. Under a matched GRPO resource envelope, FSPO reaches 66.11% held-out and 59.43% OOD accuracy, compared with 64.47% and 57.03% for PB2, the strongest evaluated adaptive baseline. Three paired training seeds give gains of +2.42 and +3.19 percentage points over the contextual bandit on held-out and OOD evaluation. Under high behavior-deployment mismatch, policy-consistent risk lowers selected-decision ECE from 0.108 to 0.053; DCTC lowers it from 0.039 to 0.022 at matched acceptance; PRCC removes false-feasible admissions on an 18-action catalog ($0.197\rightarrow0.000$); and enabling all three components reduces trajectory failure from 0.181 to 0.083 in a factorial ablation.
Related papers
- Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
- Reachability-Aware Diffusion Policy Optimization
- T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
- FARL: A Fully Automated Pipeline of RL Interface and Solver Design
- Higher-Order Action Supervision Makes A Strong Policy Class
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC
FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.02828
Explicit source versions (1)
- v2 · 2026-10-16
- Explicit source version detected · v2
Source date: 2026-10-16 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Adaptive LLM reinforcement-learning post-training changes multiple training actuators online, including rollout temperature, group size, clipping, KL regularization, verifier allocation, and update budget. Three coupled issues remain unresolved. A future-risk model trained from behavior trajectories need not estimate the risk induced by the controller that will be deployed; a score calibrated on logged state-action pairs can become miscalibrated after selective action choice; and independent per-resource minimum costs do not in general certify a feasible multi-resource continuation. We introduce FSPO, a feedback-state controller for budgeted LLM RL post-training that addresses these issues jointly. FSPO learns a policy-consistent risk-to-go model whose Bellman target follows the same frozen controller used for future decisions, together with a long-horizon utility model. Decision-conditioned trajectory calibration (DCTC) calibrates risk on cross-fitted trajectories generated by actions selected by provisional controllers. A Pareto resource continuation certificate (PRCC) admits an action only when a non-dominated cumulative reservation remains feasible over the residual horizon. Under a matched GRPO resource envelope, FSPO reaches 66.11% held-out and 59.43% OOD accuracy, compared with 64.47% and 57.03% for PB2, the strongest evaluated adaptive baseline. Three paired training seeds give gains of +2.42 and +3.19 percentage points over the contextual bandit on held-out and OOD evaluation. Under high behavior-deployment mismatch, policy-consistent risk lowers selected-decision ECE from 0.108 to 0.053; DCTC lowers it from 0.039 to 0.022 at matched acceptance; PRCC removes false-feasible admissions on an 18-action catalog ($0.197\rightarrow0.000$); and enabling all three components reduces trajectory failure from 0.181 to 0.083 in a factorial ablation.