Driving · 2026-11-03

BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents

Xuanren Chen, Chongyang Tao, Wenpeng Hu, Yong Wang, Bo Deng, Xinlei Zheng, Shuai Ma

arXiv:2611.03005PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

BLPO (Beyond-Local Policy Optimization) estimates step-level credit for long-horizon LLM agents by combining trajectory-relative advantages, local advantages over normalized actions, and task-progress advantages pooled across tasks and training updates. It summarizes trajectory prefixes into task-progress states and abstracts actions by function, using a global memory with context-relative residuals plus support/uncertainty reliability weighting. The method is critic-free and uses no additional rollouts, showing improved success on ALFWorld and WebShop.

Why it matters

Long-horizon agents need fine credit assignment, but standard step/group RL can suffer from insufficient rollout support and inconsistent interaction histories. BLPO shares experience across tasks by matching task progress and action function, while explicitly controlling bias/noise from heterogeneous contexts via return centering and reliability weighting. This aims to make step-level preferences more data-supported without adding a learned critic or extra environment interactions.

Method

  • Maps trajectory prefixes to task-progress states (execution states) and maps concrete commands to abstract action functions, enabling shared credit estimates across tasks and training updates.
  • Estimates task-progress advantages from global memory of context-relative residual returns, centering within concrete contexts and aggregating with balanced/decayed weighting.
  • Scales global advantage contrasts using effective support and uncertainty; combines with local and trajectory-level advantages in a unified critic-free, no-additional-rollouts objective.

Limitation

The paper does not state a specific limitation in the provided text.

Abstract (from arXiv)

Long-horizon tasks require large language model (LLM) agents to coordinate interdependent actions, a capability central to automating complex workflows. Stepwise group-based reinforcement learning provides finer credit than trajectory-level supervision, but reliable comparisons must account for differences in interaction history. Enforcing this consistency within the limited rollouts of each task can leave insufficient support for estimating action preferences. We propose Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. We summarize trajectory prefixes into task-progress states and abstract actions by their function, allowing corresponding decisions from different tasks to contribute to shared estimates. To control bias and noise from heterogeneous tasks, we accumulate context-relative returns in a global memory with balanced aggregation and reliability weighting. These global advantages complement local comparisons over normalized actions and trajectory-level credit without a learned critic or additional rollouts. Experiments on ALFWorld and WebShop with 1.5B and 7B policies show improved task success over the evaluated baselines under matched training budgets. Further analyses show faster learning and shorter trajectories with little computational overhead.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents

Xuanren Chen, Chongyang Tao, Wenpeng Hu, Yong Wang, Bo Deng, Xinlei Zheng, Shuai Ma

First announced by arXiv: 2026-11-03 · Original record ↗

arxiv
2611.03005
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.03005v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Long-horizon tasks require large language model (LLM) agents to coordinate interdependent actions, a capability central to automating complex workflows. Stepwise group-based reinforcement learning provides finer credit than trajectory-level supervision, but reliable comparisons must account for differences in interaction history. Enforcing this consistency within the limited rollouts of each task can leave insufficient support for estimating action preferences. We propose Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. We summarize trajectory prefixes into task-progress states and abstract actions by their function, allowing corresponding decisions from different tasks to contribute to shared estimates. To control bias and noise from heterogeneous tasks, we accumulate context-relative returns in a global memory with balanced aggregation and reliability weighting. These global advantages complement local comparisons over normalized actions and trajectory-level credit without a learned critic or additional rollouts. Experiments on ALFWorld and WebShop with 1.5B and 7B policies show improved task success over the evaluated baselines under matched training budgets. Further analyses show faster learning and shorter trajectories with little computational overhead.