Preference-Augmented Quantile-Distribution Distributional RL for Risk-Sensitive Navigation
Use distributional RL with quantile-based value modeling (e.g., IQN) and replace fixed/random scalar CVaR training with preference augmentation that shifts quantile-weight distributions. Train via preference-guided exploration over diverse risk preferences, then compute a conservative policy at deployment by optimizing against a worst-case preference distribution rather than a hand-tuned single risk scalar.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
The approach targets a reported mismatch between training-time and deployment-time risk control by learning across a broader set of tail-risk preferences. It is intended to reduce dependence on hand-tuned hyperparameters and limited external sensor signals under partial observability while improving safety-related outcomes (as claimed).
What changed
Training-time risk preference is changed from a fixed or randomly sampled scalar CVaR level to distributional preference modeling via shifted quantile-weight distributions, with preference-guided exploration. Deployment is changed from adjusting a scalar risk parameter to optimizing against a worst-case preference distribution to produce a conservative policy.
Where it helps
- Risk-sensitive autonomous navigation under uncertainty
- Training agents to handle multiple tail-risk preferences
- Conservative planning when partial observability limits reliable sensing
- Reducing reliance on manual risk-level tuning
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The source claims improvements and motivation but does not provide detailed algorithmic steps, theoretical guarantees, or reproducible specifications beyond the high-level description.
Sources
- Preference-Augmented Distributional Reinforcement Learning for Risk-Sensitive Navigation ↗
arXiv · primary · Checked 2026-11-10
