Driving · 2026-10-23
Short-to-Long: Detached Features for Long-Horizon Value Learning
Tong Che, Boris Ivanovic, Marco Pavone
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Short-to-Long (S2L) trains multiple value networks with shorter discounts and transfers their hidden features to a long-horizon behavior learner using stop-gradient (detached additions). Each source follows its own smaller-discount target while the behavior learner keeps the original long-horizon objective/discount. The authors report aggregate gains across Atari, online continuous control, and offline continuous control, including +0.049 median paired HNS on 40/57 Atari games.
Why it matters
The paper addresses non-stationary value supervision in deep RL: bootstrap targets change as policies and value networks update, and in end-to-end setups those targets also shape representation learning. S2L separates horizon-specific feature learning from the long-horizon control objective via detached feature transfer, and links the design to reduced target drift and gradient-isolated feature updates.
Method
- Train a chain of separately parameterized value learners with increasing discount horizons; each learner has its own target loss and optimization state but shares training transitions.
- Transfer same-depth hidden activations to the next (longer-horizon) learner using detached/stop-gradient additions so the receiving loss cannot backpropagate into source learners.
- Use only the final (long-horizon, base-discount) learner’s Q-values for Atari action selection or its value for continuous-control actor optimization; keep actor-critic updates consistent with the base objective.
Limitation
The authors use matched interaction budgets (not matched compute) when comparing with scaled PQN on Atari, noting that at their scale throughput is dominated by simulation rather than network evaluation.
Abstract (from arXiv)
Learning useful representations from continually changing value targets remains a challenge in deep reinforcement learning. We study the temporal horizon of this supervision: smaller discounts make these targets less sensitive to distant bootstrap changes, but using them for behavior changes the control objective. We introduce Short-to-Long (S2L), which supplies a long-horizon behavior learner with features trained by smaller-discount source networks. Detached connections prevent the behavior value loss from updating the sources, allowing each source to follow its own value objective while the behavior discount remains unchanged. Across Atari, online continuous control, and offline continuous control, the approach yields aggregate performance gains, improving the paired terminal median on 40 of 57 Atari games. Matched Atari ablations support the roles of feature transfer, smaller source discounts, and gradient isolation. Fixed-trajectory measurements further show reduced drift in source value targets with mixed discounts.
Related papers
- MAC-v2: Long-horizon model-based value expansion from pixels
- Model-Based Reinforcement Learning for Atari
- Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
- Sparse-TD-JEPA: Improving Zero-Shot Reinforcement Learning With Sparse Latent Task Representations
- From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-23 UTC
Short-to-Long: Detached Features for Long-Horizon Value Learning
Preprint posted: 2026-10-21 · Original record ↗
- arxiv
- 2610.30461
Explicit source versions (1)
- v1 · 2026-10-21
- Preprint available
Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Learning useful representations from continually changing value targets remains a challenge in deep reinforcement learning. We study the temporal horizon of this supervision: smaller discounts make these targets less sensitive to distant bootstrap changes, but using them for behavior changes the control objective. We introduce Short-to-Long (S2L), which supplies a long-horizon behavior learner with features trained by smaller-discount source networks. Detached connections prevent the behavior value loss from updating the sources, allowing each source to follow its own value objective while the behavior discount remains unchanged. Across Atari, online continuous control, and offline continuous control, the approach yields aggregate performance gains, improving the paired terminal median on 40 of 57 Atari games. Matched Atari ablations support the roles of feature transfer, smaller source discounts, and gradient isolation. Fixed-trajectory measurements further show reduced drift in source value targets with mixed discounts.
