Driving · 2026-11-06

The Blindfolded Controller: Decision Model-Guided Training for Reinforcement Learning

Muhammad Ibrahim Khan, Aniket Dixit, Faizan Ahmed, James Brusey

arXiv:2611.07934PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose Decision Model-Guided Training (DMGT): use a decision model (DM) queried with natural-language descriptions of the environment/task to label actions, imitate those decisions to initialise an RL policy (BC/DAgger), then use PPO to improve from reward. Experiments on seven tasks show DMGT outperforms PPO alone in all seven and DM alone in five.

Why it matters

DMGT aims to reduce sparse-reward/exploration difficulty by starting RL from useful, zero-shot behaviour produced by a decision model that “is blindfolded to the reward.” The paper also studies how to write DM queries via ablation across seven tasks to derive guidelines for environment/task descriptions.

Method

  • Use a decision model (DM) queried with natural-language state/task/environment description (reward not included) to output an action each step (DM policy).
  • Train an RL policy via imitation learning: behaviour cloning from DM-labelled rollouts, optionally refined with DAgger by labeling states visited by the student.
  • Initialise PPO with the best imitation-trained student weights, then train PPO with default hyperparameters to improve from reward (DM used only before PPO begins).
Abstract (from arXiv)

Reinforcement learning (RL) provides a framework for continuous improvement through interaction with an environment. However, RL agents struggle against tasks that offer exploration challenges and sparse rewards, because learning efficient policies from scratch is slow and sample-inefficient. In contrast, decision models (DMs), given a description of the environment and task, can perform well in sparse-reward tasks, but, as they do not learn from reward, they are not able to improve towards the optimal solution. In this paper, we present Decision Model-Guided Training (DMGT), a technique that uses a DM with a description of the environment and task to initialise an RL policy, combining the DM's ability to act without training with RL's ability to improve from reward. Instead of exploring from scratch, the agent reaches the goal and receives reward from its first episodes, and RL then improves its policy beyond what the DM achieves. We demonstrate through experiments on seven tasks that our approach outperforms RL in all seven and DM in five. In addition, we provide comprehensive guidelines, derived from ablation studies, for describing an environment and task to a DM.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-06 · Indexed by Pub-AI: 2026-11-06 UTC

preprint

The Blindfolded Controller: Decision Model-Guided Training for Reinforcement Learning

Muhammad Ibrahim Khan, Aniket Dixit, Faizan Ahmed, James Brusey

First announced by arXiv: 2026-11-06 · Original record ↗

arxiv
2611.07934
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-06 · First detected: 2026-11-06 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.07934v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Reinforcement learning (RL) provides a framework for continuous improvement through interaction with an environment. However, RL agents struggle against tasks that offer exploration challenges and sparse rewards, because learning efficient policies from scratch is slow and sample-inefficient. In contrast, decision models (DMs), given a description of the environment and task, can perform well in sparse-reward tasks, but, as they do not learn from reward, they are not able to improve towards the optimal solution. In this paper, we present Decision Model-Guided Training (DMGT), a technique that uses a DM with a description of the environment and task to initialise an RL policy, combining the DM's ability to act without training with RL's ability to improve from reward. Instead of exploring from scratch, the agent reaches the goal and receives reward from its first episodes, and RL then improves its policy beyond what the DM achieves. We demonstrate through experiments on seven tasks that our approach outperforms RL in all seven and DM in five. In addition, we provide comprehensive guidelines, derived from ablation studies, for describing an environment and task to a DM.