Driving · 2026-10-16
Progress Reward Model for Reinforcement Learning via Large Language Models
Xiuhui Zhang, Ning Gao, Xingyu Jiang, Yihui Chen, Yuheng Pan, Mohan Zhang, Yue Deng
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
PRM4RL integrates LLM-based task decomposition with a Potential-based Progress Reward Model. The LLM generates a subtask list plus a state-based subtask determination function and fine-grained progress functions per subtask. The overall progress Φ(s) is used to build a dense reward via a temporal-difference/potential-based form, with policy invariance and convergence guarantees. Experiments on Metaworld and Maniskill show improved success rate and reduced LLM calls; code is released.
Why it matters
The authors target long-term, sparse-reward robotics control by combining high-level LLM planning (via subtask priors and low-cost subtask determination) with low-level dense reward (derived from progress as a potential). They provide theoretical claims (policy invariance/optimality and convergence guarantees for potential-based shaping) and report empirical superiority over LLM-planning and LLM-reward methods.
Method
- Use an LLM to decompose a long-term task into subtasks (plan_list) with natural-language subtask priors, and generate an efficient state-only subtask determination function Ψ(s) to select the current subtask without calling the LLM each timestep.
- For each subtask, define a fine-grained progress function and integrate them into an overall progress Φ(s); construct PRM reward using the temporal-difference form R_t^PRM = γ·Φ(s_{t+1}) − Φ(s_t) plus an indicator-weighted success/bonus term.
- Augment RL by embedding subtask priors (via SimCSE) into the policy state space and training with PRM rewards, using SAC or PPO; theoretical results claim policy invariance and convergence guarantees for potential-based shaping.
Abstract (from arXiv)
Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at https://github.com/deng-ai-lab/PRM4RL.
Related papers
- Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond
- TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
- FARL: A Fully Automated Pipeline of RL Interface and Solver Design
- AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
- TD-MPC2: Scalable, Robust World Models for Continuous Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-15 · Indexed by Pub-AI: 2026-10-16 UTC
Progress Reward Model for Reinforcement Learning via Large Language Models
Preprint posted: 2026-10-15 · Original record ↗
- arxiv
- 2610.22056
Explicit source versions (1)
- v1 · 2026-10-15
- Preprint available
Source date: 2026-10-15 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at https://github.com/deng-ai-lab/PRM4RL.