Driving · 2026-10-23

Short-to-Long: Detached Features for Long-Horizon Value Learning

Tong Che, Boris Ivanovic, Marco Pavone

arXiv:2610.30461PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Short-to-Long (S2L) trains multiple value networks with shorter discounts and transfers their hidden features to a long-horizon behavior learner using stop-gradient (detached additions). Each source follows its own smaller-discount target while the behavior learner keeps the original long-horizon objective/discount. The authors report aggregate gains across Atari, online continuous control, and offline continuous control, including +0.049 median paired HNS on 40/57 Atari games.

Why it matters

The paper addresses non-stationary value supervision in deep RL: bootstrap targets change as policies and value networks update, and in end-to-end setups those targets also shape representation learning. S2L separates horizon-specific feature learning from the long-horizon control objective via detached feature transfer, and links the design to reduced target drift and gradient-isolated feature updates.

Method

  • Train a chain of separately parameterized value learners with increasing discount horizons; each learner has its own target loss and optimization state but shares training transitions.
  • Transfer same-depth hidden activations to the next (longer-horizon) learner using detached/stop-gradient additions so the receiving loss cannot backpropagate into source learners.
  • Use only the final (long-horizon, base-discount) learner’s Q-values for Atari action selection or its value for continuous-control actor optimization; keep actor-critic updates consistent with the base objective.

Limitation

The authors use matched interaction budgets (not matched compute) when comparing with scaled PQN on Atari, noting that at their scale throughput is dominated by simulation rather than network evaluation.

Abstract (from arXiv)

Learning useful representations from continually changing value targets remains a challenge in deep reinforcement learning. We study the temporal horizon of this supervision: smaller discounts make these targets less sensitive to distant bootstrap changes, but using them for behavior changes the control objective. We introduce Short-to-Long (S2L), which supplies a long-horizon behavior learner with features trained by smaller-discount source networks. Detached connections prevent the behavior value loss from updating the sources, allowing each source to follow its own value objective while the behavior discount remains unchanged. Across Atari, online continuous control, and offline continuous control, the approach yields aggregate performance gains, improving the paired terminal median on 40 of 57 Atari games. Matched Atari ablations support the roles of feature transfer, smaller source discounts, and gradient isolation. Fixed-trajectory measurements further show reduced drift in source value targets with mixed discounts.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-23 UTC

preprint

Short-to-Long: Detached Features for Long-Horizon Value Learning

Tong Che, Boris Ivanovic, Marco Pavone

Preprint posted: 2026-10-21 · Original record ↗

arxiv
2610.30461
Explicit source versions (1)
  • v1 · 2026-10-21
  1. Preprint available

    Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Learning useful representations from continually changing value targets remains a challenge in deep reinforcement learning. We study the temporal horizon of this supervision: smaller discounts make these targets less sensitive to distant bootstrap changes, but using them for behavior changes the control objective. We introduce Short-to-Long (S2L), which supplies a long-horizon behavior learner with features trained by smaller-discount source networks. Detached connections prevent the behavior value loss from updating the sources, allowing each source to follow its own value objective while the behavior discount remains unchanged. Across Atari, online continuous control, and offline continuous control, the approach yields aggregate performance gains, improving the paired terminal median on 40 of 57 Atari games. Matched Atari ablations support the roles of feature transfer, smaller source discounts, and gradient isolation. Fixed-trajectory measurements further show reduced drift in source value targets with mixed discounts.