Driving · 2026-10-08
Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving
Mahmoud Selim, Cristina Cipriani, Karl Henrik Johansson
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce ICDP, an offline RL method for autonomous driving that explicitly controls “interaction distribution shift” (IDS), where an ego trajectory may be individually supported yet poorly supported jointly with surrounding-agent behavior. ICDP decomposes joint-support degradation into an ego component and an interaction component, estimates the interaction part via contrastive density-ratio estimation, and uses it to regularize diffusion-policy optimization. Evaluations are reported on nuPlan, Interplan, and real-world truck experiments.
Why it matters
In interactive driving logs, controlling distribution shift only in the ego action space can miss cases where the ego plan is compatible marginally but incompatible with surrounding-agent evolution. ICDP targets this by constraining policy improvement at the level of multi-agent interaction support, without requiring joint-density modeling, agent-response prediction, or rollouts in reactive simulators/world models during optimization.
Method
- ICDP trains a diffusion actor for candidate ego trajectories and a chunk-level critic ensemble (with a lower-confidence-bound estimate) for long-horizon value evaluation.
- The authors define joint-support degradation and show it decomposes into ego-support shift and a residual interaction-support term (IDS).
- IDS (interaction-support term) is recovered contrastively using two classifiers (joint ego–agent vs ego-only) and the residual score g_I=g_J−g_E is used as an interaction-support regularizer during diffusion-policy optimization.
Abstract (from arXiv)
Offline reinforcement learning enables reward-driven policy improvement from fixed datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data, rendering value estimates unreliable. Existing approaches primarily control this shift in the policy's own action space. In interactive environments such as autonomous driving, this can be insufficient: a candidate ego trajectory may remain well supported under the marginal behavior distribution while being poorly supported jointly with the surrounding-agent behavior observed in the logged interaction. We refer to this degradation in interaction support as \emph{interaction distribution shift} (IDS), and introduce \emph{Interaction-Constrained Drive Policy} (ICDP), an offline reinforcement learning framework that explicitly controls interaction-level distribution shift. Starting from the joint data distribution over ego and surrounding-agent futures, we show that joint-support degradation decomposes exactly into an ego-support component and a residual interaction-support component. We recover the latter through contrastive density-ratio estimation, isolating interaction compatibility without explicit joint-density modeling, surrounding-agent prediction, or rollouts in reactive simulators or learned world models during policy optimization. Closed-loop evaluations on nuPlan, Interplan and real-world truck experiments show that ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios. Project webpage: https://mahmoud-selim.github.io/ICDP/
Related papers
- RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts
- ReWAM: Reciprocal World Action Models for Interactive Autonomous Driving
- In-Distribution Imagination for Model-Based Offline Reinforcement Learning
- Closed-Loop Aware Planning via Imitation-Free Speed Scoring for End-to-End Autonomous Driving
- Higher-Order Action Supervision Makes A Strong Policy Class
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC
Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving
Preprint posted: 2026-10-07 · Original record ↗
- arxiv
- 2610.09763
Explicit source versions (1)
- v2 · 2026-10-12
- Explicit source version detected · v2
Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Preprint available
Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Offline reinforcement learning enables reward-driven policy improvement from fixed datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data, rendering value estimates unreliable. Existing approaches primarily control this shift in the policy's own action space. In interactive environments such as autonomous driving, this can be insufficient: a candidate ego trajectory may remain well supported under the marginal behavior distribution while being poorly supported jointly with the surrounding-agent behavior observed in the logged interaction. We refer to this degradation in interaction support as \emph{interaction distribution shift} (IDS), and introduce \emph{Interaction-Constrained Drive Policy} (ICDP), an offline reinforcement learning framework that explicitly controls interaction-level distribution shift. Starting from the joint data distribution over ego and surrounding-agent futures, we show that joint-support degradation decomposes exactly into an ego-support component and a residual interaction-support component. We recover the latter through contrastive density-ratio estimation, isolating interaction compatibility without explicit joint-density modeling, surrounding-agent prediction, or rollouts in reactive simulators or learned world models during policy optimization. Closed-loop evaluations on nuPlan, Interplan and real-world truck experiments show that ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios. Project webpage: https://mahmoud-selim.github.io/ICDP/