Driving · 2026-11-06
The Blindfolded Controller: Decision Model-Guided Training for Reinforcement Learning
Muhammad Ibrahim Khan, Aniket Dixit, Faizan Ahmed, James Brusey
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose Decision Model-Guided Training (DMGT): use a decision model (DM) queried with natural-language descriptions of the environment/task to label actions, imitate those decisions to initialise an RL policy (BC/DAgger), then use PPO to improve from reward. Experiments on seven tasks show DMGT outperforms PPO alone in all seven and DM alone in five.
Why it matters
DMGT aims to reduce sparse-reward/exploration difficulty by starting RL from useful, zero-shot behaviour produced by a decision model that “is blindfolded to the reward.” The paper also studies how to write DM queries via ablation across seven tasks to derive guidelines for environment/task descriptions.
Method
- Use a decision model (DM) queried with natural-language state/task/environment description (reward not included) to output an action each step (DM policy).
- Train an RL policy via imitation learning: behaviour cloning from DM-labelled rollouts, optionally refined with DAgger by labeling states visited by the student.
- Initialise PPO with the best imitation-trained student weights, then train PPO with default hyperparameters to improve from reward (DM used only before PPO begins).
Abstract (from arXiv)
Reinforcement learning (RL) provides a framework for continuous improvement through interaction with an environment. However, RL agents struggle against tasks that offer exploration challenges and sparse rewards, because learning efficient policies from scratch is slow and sample-inefficient. In contrast, decision models (DMs), given a description of the environment and task, can perform well in sparse-reward tasks, but, as they do not learn from reward, they are not able to improve towards the optimal solution. In this paper, we present Decision Model-Guided Training (DMGT), a technique that uses a DM with a description of the environment and task to initialise an RL policy, combining the DM's ability to act without training with RL's ability to improve from reward. Instead of exploring from scratch, the agent reaches the goal and receives reward from its first episodes, and RL then improves its policy beyond what the DM achieves. We demonstrate through experiments on seven tasks that our approach outperforms RL in all seven and DM in five. In addition, we provide comprehensive guidelines, derived from ablation studies, for describing an environment and task to a DM.
Related papers
- FARL: A Fully Automated Pipeline of RL Interface and Solver Design
- Can Jev be Your Q or Policy in Reinforcement Learning?
- Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond
- Progress Reward Model for Reinforcement Learning via Large Language Models
- Statistical Foundations of Reinforcement Learning: A Non-Asymptotic Perspective
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-06 · Indexed by Pub-AI: 2026-11-06 UTC
The Blindfolded Controller: Decision Model-Guided Training for Reinforcement Learning
First announced by arXiv: 2026-11-06 · Original record ↗
- arxiv
- 2611.07934
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-06 · First detected: 2026-11-06 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.07934v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Reinforcement learning (RL) provides a framework for continuous improvement through interaction with an environment. However, RL agents struggle against tasks that offer exploration challenges and sparse rewards, because learning efficient policies from scratch is slow and sample-inefficient. In contrast, decision models (DMs), given a description of the environment and task, can perform well in sparse-reward tasks, but, as they do not learn from reward, they are not able to improve towards the optimal solution. In this paper, we present Decision Model-Guided Training (DMGT), a technique that uses a DM with a description of the environment and task to initialise an RL policy, combining the DM's ability to act without training with RL's ability to improve from reward. Instead of exploring from scratch, the agent reaches the goal and receives reward from its first episodes, and RL then improves its policy beyond what the DM achieves. We demonstrate through experiments on seven tasks that our approach outperforms RL in all seven and DM in five. In addition, we provide comprehensive guidelines, derived from ablation studies, for describing an environment and task to a DM.
