Unassigned · 2026-10-21
TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
Yi Wen, Hao Chen, Wanyu Wang, Maolin Wang, Pengyue Jia, Derong Xu, Huize Tan, Yingyi Zhang, Wenlin Zhang, Weihong Luo, Xiku Du, Xiangyu Zhao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TrajEvolve proposes trajectory evolution for long-horizon reinforcement learning using hindsight credit assignment. It reformulates step-level credit as a binary “necessary vs redundant” classification, obtains step labels via environment replay (hindsight verification), trains a discriminator, and uses redundancy-aware reward shaping plus step-level advantage adjustment. It also generates “evolutionary trajectories” by refining historical trajectories (redundancy elimination, format correction, error rollback) and re-executes them for updated training rewards.
Why it matters
In long-horizon RL with sparse final rewards, existing methods update steps indiscriminately and can reinforce redundant steps. The authors aim to actively optimize trajectories by (1) making credit assignment verifiable via binary necessity labels and counterfactual execution, and (2) training on evolved, non-redundant trajectories that may not be sampled by the current policy under practical training budgets.
Method
- Reformulate step-level credit into binary necessity classification using a discriminator Dϕ that predicts whether each step is necessary (ŷt=1) or redundant (ŷt=0).
- Obtain ground-truth necessity labels via hindsight verification on environment-changing steps by counterfactual execution; train the discriminator with cross-entropy loss on verified trajectories.
- Use redundancy-aware reward shaping and step-level advantage adjustment from discriminator outputs; further refine historical trajectories with redundancy elimination, format correction, or error rollback, then re-execute evolved trajectories for updated rewards.
Limitation
The provided text does not state specific limitations of the approach.
Abstract (from arXiv)
The Reinforcement Learning (RL) paradigm has achieved remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). However, when applied to long-horizon tasks, it faces severe challenges: i) Due to the high task complexity, the model struggles to sample enough successful trajectories, resulting in slow convergence. ii) Relying solely on sparse final rewards causes all steps to be updated indiscriminately, encouraging redundant step generation. When both the quantity and quality of successful trajectories become difficult to guarantee, the performance is severely constrained. This dilemma stems from the limitations: existing RL paradigms rely entirely on the reasoning capability of base models, lacking the ability to actively optimize trajectories. To address the above issues, we propose a novel trajectory evolution paradigm with hindsight credit assignment, termed TrajEvolve. TrajEvolve not only relies on the trajectory inferred by the model but also generates ``evolutionary trajectories'' by refining low-quality steps in historical trajectories, thereby exposing the policy to high-quality compositions that lie within the base model's support but are rarely sampled within practical training budgets. Furthermore, TrajEvolve reformulates step-level continuous credit assignment as a binary classification task, whether each step is necessary for task completion. This simplification not only reduces the learning difficulty but also enables step labels to be obtained through counterfactual verification, achieving precise identification of redundant steps. Extensive experiments on two datasets demonstrate the superiority of the proposed method.
Related papers
- T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
- Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
- Progress Reward Model for Reinforcement Learning via Large Language Models
- Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond
- Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC
TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28060
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · OAI arXiv created; not header datestamp (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
The Reinforcement Learning (RL) paradigm has achieved remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). However, when applied to long-horizon tasks, it faces severe challenges: i) Due to the high task complexity, the model struggles to sample enough successful trajectories, resulting in slow convergence. ii) Relying solely on sparse final rewards causes all steps to be updated indiscriminately, encouraging redundant step generation. When both the quantity and quality of successful trajectories become difficult to guarantee, the performance is severely constrained. This dilemma stems from the limitations: existing RL paradigms rely entirely on the reasoning capability of base models, lacking the ability to actively optimize trajectories. To address the above issues, we propose a novel trajectory evolution paradigm with hindsight credit assignment, termed TrajEvolve. TrajEvolve not only relies on the trajectory inferred by the model but also generates ``evolutionary trajectories'' by refining low-quality steps in historical trajectories, thereby exposing the policy to high-quality compositions that lie within the base model's support but are rarely sampled within practical training budgets. Furthermore, TrajEvolve reformulates step-level continuous credit assignment as a binary classification task, whether each step is necessary for task completion. This simplification not only reduces the learning difficulty but also enables step labels to be obtained through counterfactual verification, achieving precise identification of redundant steps. Extensive experiments on two datasets demonstrate the superiority of the proposed method.