Driving · 2026-10-14
TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training
Yuqian Fu, Kaiwen Jiang, Zijun Song, Jiawei Wang, Yuanheng Zhu, Dongbin Zhao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TSRL (Trigger-based Supervised Reinforcement Learning) uses RL as the optimization backbone and activates expert SFT only on “hard states” where the policy fails, detected via Group Success Rate (GSR) below a threshold. The authors theoretically bound forgetting by restricting supervision to failure states, and report consistent gains across ALFWorld, WebShop, and Deep Search at 7B and 32B scales, with up to 9.6 points more prior capability retention than SFT→RL baselines.
Why it matters
Catastrophic forgetting in continual agent training creates a trade-off: SFT is sample-efficient but can cause severe forgetting, while RL preserves prior capability but is slower and less guided. TSRL targets this trade-off by making supervised updates sparse and localized to failure states while keeping the main optimization on-policy, and the authors report better retention and competitive target-task performance.
Method
- Use group-based RL (e.g., GRPO/DAPO) as the RL optimization backbone, while optionally adding SFT gradients.
- Define a trigger using Group Success Rate (GSR); if GSR(s) ≤ δ, the method queries the expert and applies an SFT loss only on those triggered on-policy states.
- Apply temporal annealing to linearly decay the expert-supervision influence λ(t) to 0 over training.
Limitation
The text does not state explicit limitations.
Abstract (from arXiv)
Large language model (LLM) agents excel at complex, multi-step tasks but suffer from catastrophic forgetting during continual learning--acquiring new skills often degrades previously mastered capabilities. Existing post-training paradigms face a fundamental trade-off: Supervised Fine-Tuning (SFT) is sample-efficient but prone to severe forgetting, while Reinforcement Learning (RL) preserves prior capabilities but converges slowly without expert guidance. To bridge this gap, we introduce Trigger-based Supervised Reinforcement Learning (TSRL), a framework that integrates the strengths of both paradigms through an adaptive, state-dependent expert supervision mechanism. TSRL employs RL as the optimization backbone, activating SFT selectively on "hard states" where the policy fails. Theoretically, we show that restricting supervision to failure states derives an upper bound on the cumulative parameter updates, thereby reducing forgetting compared to full SFT. Empirically, TSRL achieves consistent gains across three agent benchmarks (ALFWorld, WebShop, and Deep Search) at both 7B and 32B scales. In continual learning scenarios, our method retains up to 9.6 points more capability on prior tasks than SFT->RL baselines, while matching or exceeding target-task performance.
Related papers
- Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents
- Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
- T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
- AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
- QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-14 UTC
TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training
Preprint posted: 2026-10-13 · Original record ↗
- arxiv
- 2610.17726
Explicit source versions (1)
- v1 · 2026-10-13
- Preprint available
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Large language model (LLM) agents excel at complex, multi-step tasks but suffer from catastrophic forgetting during continual learning--acquiring new skills often degrades previously mastered capabilities. Existing post-training paradigms face a fundamental trade-off: Supervised Fine-Tuning (SFT) is sample-efficient but prone to severe forgetting, while Reinforcement Learning (RL) preserves prior capabilities but converges slowly without expert guidance. To bridge this gap, we introduce Trigger-based Supervised Reinforcement Learning (TSRL), a framework that integrates the strengths of both paradigms through an adaptive, state-dependent expert supervision mechanism. TSRL employs RL as the optimization backbone, activating SFT selectively on "hard states" where the policy fails. Theoretically, we show that restricting supervision to failure states derives an upper bound on the cumulative parameter updates, thereby reducing forgetting compared to full SFT. Empirically, TSRL achieves consistent gains across three agent benchmarks (ALFWorld, WebShop, and Deep Search) at both 7B and 32B scales. In continual learning scenarios, our method retains up to 9.6 points more capability on prior tasks than SFT->RL baselines, while matching or exceeding target-task performance.