Unassigned · 2026-10-14
FARL: A Fully Automated Pipeline of RL Interface and Solver Design
Tianyu Jin, Liuyu Xiang, Chaowen Yang, Xingyang Chang, Pei Pei Li, Zhaofeng He
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
FARL automates RL design by jointly searching observation and reward programs, learning algorithms, and training settings, using training statistics and observed behavior to diagnose failures and test specific changes. A diversity-preserving search uses short trials first, and scientific memory records each finding with its test conditions for later reuse. On expert-withheld and expert-available navigation, locomotion, and manipulation tasks, FARL outperforms evaluated automated baselines.
Why it matters
The paper targets a key bottleneck in applying RL to new tasks: human expertise is still needed to design the learning interface (observations/rewards) and the solver/training pipeline. FARL’s closed-loop design–train–diagnose framework explicitly handles coupled design choices, using controlled comparisons and recorded test conditions to revise unsuccessful components.
Method
- Joint interface–solver co-design: FARL proposes observation programs, reward programs, and compatible solver settings (learner, hyperparameters, exploration, and training budget) from task API/evaluator plus evidence from earlier experiments.
- Diversity-preserving successive-halving-style search: it screens designs with short budgets, retains candidates for competitiveness/learning potential/experimental value, and preserves components for later diagnostic combinations.
- Falsifiable diagnosis with controlled tests: an LLM uses learning curves, optimizer statistics, and milestone/trajectory behavior to propose changes and predictions, then tests them by specifying changed vs fixed components and training budgets, recording outcomes in scientific memory.
Abstract (from arXiv)
Existing reinforcement learning (RL) largely focuses on optimizing learning algorithms within a pre-defined task formulation, while applying RL to new tasks still relies heavily on human expertise to design the learning pipeline: what an agent observes, how it is rewarded, which algorithm it uses, and how training is configured. Automating these choices is challenging because they are tightly coupled. We introduce FARL (Fully Automated Reinforcement Learning), a framework that automates RL design by jointly searching observation and reward programs, learning algorithms, and training settings. FARL uses training statistics and observed behavior to diagnose failures, propose changes, and test whether they improve learning. Its search preserves diverse candidates and the controls needed for these tests, using a staged budget to screen designs before investing in longer training. Experimental memory records each finding together with its conditions, so an unsuccessful component can be reconsidered in a different combination. Across expert-withheld and expert-available settings, FARL outperforms the evaluated automated baselines on challenging navigation, locomotion, and manipulation tasks. On Adroit Hand Relocate, FARL achieves 97.1% success at 5M training steps per policy, compared with 0% for Eureka and LiMEN and 1.0% for native Proximal Policy Optimization (PPO) under the same policy training budget.
Related papers
- Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design
- Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond
- AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
- ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation
- CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-11 · Indexed by Pub-AI: 2026-10-14 UTC
FARL: A Fully Automated Pipeline of RL Interface and Solver Design
Preprint posted: 2026-10-11 · Original record ↗
- arxiv
- 2610.15691
Explicit source versions (1)
- v1 · 2026-10-11
- Preprint available
Source date: 2026-10-11 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Existing reinforcement learning (RL) largely focuses on optimizing learning algorithms within a pre-defined task formulation, while applying RL to new tasks still relies heavily on human expertise to design the learning pipeline: what an agent observes, how it is rewarded, which algorithm it uses, and how training is configured. Automating these choices is challenging because they are tightly coupled. We introduce FARL (Fully Automated Reinforcement Learning), a framework that automates RL design by jointly searching observation and reward programs, learning algorithms, and training settings. FARL uses training statistics and observed behavior to diagnose failures, propose changes, and test whether they improve learning. Its search preserves diverse candidates and the controls needed for these tests, using a staged budget to screen designs before investing in longer training. Experimental memory records each finding together with its conditions, so an unsuccessful component can be reconsidered in a different combination. Across expert-withheld and expert-available settings, FARL outperforms the evaluated automated baselines on challenging navigation, locomotion, and manipulation tasks. On Adroit Hand Relocate, FARL achieves 97.1% success at 5M training steps per policy, compared with 0% for Eureka and LiMEN and 1.0% for native Proximal Policy Optimization (PPO) under the same policy training budget.