Driving · 2026-10-19
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu, Saiyong Yang, Xiangyang Ji
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TreeOPSD is a critic-free, tree-aware posterior self-distillation method for long-horizon agentic RL. It treats each interaction turn as a node in a rollout tree, branches alternative continuations, and backs up terminal verifier rewards into implicit turn-level credit. It maps this credit to a closed-form bounded teacher distribution (relative to a frozen behavior policy) and then uses Forward-KL posterior distillation across sequential mini-batches so the gradient vanishes at the target and reverses after overshoot.
Why it matters
The authors target two RLVR training issues: sparse/high-variance turn credit assignment from terminal rewards, and policy staleness/overshooting from sequential mini-batch updates on a fixed rollout batch. TreeOPSD claims to derive dense turn-level credit directly from terminal verifier rewards without a learned critic or process reward model, and to stabilize off-policy sequential updates via a finite, self-correcting teacher target built from backed-up tree credit.
Method
- Build initialize-then-expand rollout trees from a frozen behavior policy; standardize terminal rewards into trajectory advantages and recursively back up outcomes to assign turn-level credit on shared-prefix tree nodes (critic-free).
- Convert backed-up turn credit V_i into a closed-form bounded Bernoulli teacher distribution q_i using an affine map calibrated by a temperature τ to keep targets in a finite range.
- Across sequential mini-batches, re-evaluate the student posterior p_{θ,i} and minimize Forward-KL between frozen teacher Q_i* and current student, claiming the tangent recovers the exact tree policy-gradient direction and the residual vanishes at the finite target then reverses sign after overshoot.
Abstract (from arXiv)
Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing approaches obtain denser credit via learned critics, process reward models, or on-policy distillation, but they rely on a value network, step-level annotations, or a stronger external teacher that increase training complexity. Meanwhile, importance sampling and clipping mitigate policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each credit to a closed-form, bounded teacher distribution parameterized by the likelihood ratio relative to the frozen behavior policy. Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting.
Related papers
- Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents
- T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
- Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards
- QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents
- Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-16 · Indexed by Pub-AI: 2026-10-19 UTC
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Preprint posted: 2026-10-16 · Original record ↗
- arxiv
- 2610.23181
Explicit source versions (1)
- v1 · 2026-10-16
- Preprint available
Source date: 2026-10-16 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing approaches obtain denser credit via learned critics, process reward models, or on-policy distillation, but they rely on a value network, step-level annotations, or a stronger external teacher that increase training complexity. Meanwhile, importance sampling and clipping mitigate policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each credit to a closed-form, bounded teacher distribution parameterized by the likelihood ratio relative to the frozen behavior policy. Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting.