Unassigned · 2026-10-02
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Bo-Wen Zhang, Junwei He, Maoqi Liu, Feiran Li, Song-Lin Lv, Wentao Ma, Rongyi Lin, Shuhan Zhong, Lan-Zhe Guo
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
T2SPO (Trajectory-to-Step Policy Optimization) uses successful past trajectories to provide step-level feedback for LLM-agent RL without training a critic. It builds a context of state representations paired with observed remaining turns-to-success, uses a frozen TabPFN regressor to estimate remaining distance during new rollouts, and turns changes in predicted remaining distance into auxiliary step credit added to GRPO advantages. Experiments on ALFWorld and WebShop with 1.5B/7B models report higher task success than GRPO.
Why it matters
Many interactive tasks only provide sparse final outcomes, making credit assignment across intermediate steps difficult. The authors propose reusing successful trajectories to estimate progress (remaining distance to success) and convert it into step-level auxiliary credit, aiming to improve policy learning while retaining GRPO’s outcome-based supervision and avoiding training an additional value model/critic.
Method
- Derive supervision from successful trajectories by pairing each visited decision state with the number of turns remaining along its own successful continuation; unsuccessful states are excluded (success-only).
- Use a frozen TabPFN regressor conditioned on a shared historical context (state representations + remaining-length labels) to predict remaining distance at states in new rollouts; refresh context after completed trajectories are scored.
- Convert adjacent-state potential differences in predicted remaining distance into auxiliary step credit, then augment GRPO’s outcome-based advantages (or optionally add as auxiliary rewards in PPO).
Limitation
The authors note that normalization, gating, and terminal exceptions modify the potential difference, so the standard policy-invariance guarantee does not extend to the full T2SPO procedure.
Abstract (from arXiv)
Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However, rewards in many interactive tasks reflect only the final outcome, providing limited guidance on which intermediate decisions advance the task. Successful training trajectories contain intermediate states that can provide supervision for subsequent interactions. We introduce Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning. T2SPO derives remaining-distance targets from successful trajectories and pairs them with representations of the states visited along the way. Conditioned on these examples, a pretrained TabPFN regressor estimates the remaining distance to success at each state of a new rollout. Changes in this distance estimate across consecutive states yield auxiliary credit for agent steps alongside task-level supervision. As training proceeds, newly completed trajectories refresh the estimator's context, incorporating new experience without updating its parameters. Experiments with 1.5B and 7B language models on ALFWorld and WebShop show that T2SPO consistently improves overall task success over GRPO.
Related papers
- TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
- Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
- Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
- TD-MPC2: Scalable, Robust World Models for Continuous Control
- Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.00388
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However, rewards in many interactive tasks reflect only the final outcome, providing limited guidance on which intermediate decisions advance the task. Successful training trajectories contain intermediate states that can provide supervision for subsequent interactions. We introduce Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning. T2SPO derives remaining-distance targets from successful trajectories and pairs them with representations of the states visited along the way. Conditioned on these examples, a pretrained TabPFN regressor estimates the remaining distance to success at each state of a new rollout. Changes in this distance estimate across consecutive states yield auxiliary credit for agent steps alongside task-level supervision. As training proceeds, newly completed trajectories refresh the estimator's context, incorporating new experience without updating its parameters. Experiments with 1.5B and 7B language models on ALFWorld and WebShop show that T2SPO consistently improves overall task success over GRPO.