Driving · 2026-11-10

DPDP: Diffusion Policy-Steered Diffusion Planner for Offline Reinforcement Learning

Ruyi An, Mingyuan Zhou

arXiv:2611.10993PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DPDP combines a diffusion planner with a value-optimized diffusion policy at the planner’s execution step in offline RL. The planner proposes a next-state transition; a diffusion inverse-dynamics executor samples an action as a Bayesian posterior that keeps the planned transition likelihood while replacing the behavior prior with a critic-guided policy (DPDP-inf via training-free score composition/correction, DPDP-ft via one executor fine-tuning).

Why it matters

The authors target a specific gap: diffusion policies are value-aware but do not commit to intermediate states, while diffusion planners are far-sighted but execute only via behavior support. DPDP carries value information into the exact step where a planned transition is turned into an action, aiming to improve execution without discarding the planner’s proposed transition.

Method

  • Cast inverse dynamics as a conditional diffusion posterior p_I(a|s,g) with a diffusion null branch p_B(a|s) representing behavioral prior, using CFG training to learn a transition-conditioned branch and a state-conditioned null branch.
  • Replace the behavior prior in the executor posterior with a critic-trained diffusion policy p_Q(a|s), preserving the planner’s transition likelihood p(g|s,a) while tilting actions toward higher-value behavior.
  • Realize the unified posterior in two ways: DPDP-inf composes frozen diffusion scores at matched noise levels with Langevin correction; DPDP-ft amortizes the preference via fine-tuning the inverse-dynamics executor with a preference objective.

Limitation

The authors state that their finite-step implementation targets the composed clean-action distribution approximately (they do not claim exact clean-endpoint sampling at finite cost).

Abstract (from arXiv)

In offline reinforcement learning, diffusion models fill two roles with complementary strengths. Typically guided by a critic, a diffusion policy is value-aware and favors high-return, behavior-supported actions, but it is short-sighted: it never lays out the states it intends to visit. A diffusion planner is far-sighted, synthesizing such a long-horizon trajectory, but it is only behavior-driven at execution, realizing each planned transition using only offline behavior.Each is strong exactly where the other is weak. We introduce DPDP to bring them together at the planner's execution step, where a planned transition becomes an action. Reading the planner's inverse-dynamics executor as a posterior over actions: the environment's transition likelihood relates an action to the planned next state, while the behavioral prior determines its weight under the offline data. Replacing this prior with a value-optimized diffusion policy preserves the planned transition likelihood and favors valuable actions that can serve the plan. We target the resulting clean-action distribution in two ways. A training-free sampler composes policy and inverse-dynamics scores at matched action noise levels and applies Langevin correction; this route uses a shared action noise schedule and an approximately common terminal prior. A preference objective instead amortizes the transfer into a single diffusion executor. Across D4RL, DPDP improves the planner it builds on, staying competitive with reactive diffusion policies on locomotion and leading on long-horizon navigation.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

DPDP: Diffusion Policy-Steered Diffusion Planner for Offline Reinforcement Learning

Ruyi An, Mingyuan Zhou

First announced by arXiv: 2026-11-10 · Original record ↗

arxiv
2611.10993
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.10993v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

In offline reinforcement learning, diffusion models fill two roles with complementary strengths. Typically guided by a critic, a diffusion policy is value-aware and favors high-return, behavior-supported actions, but it is short-sighted: it never lays out the states it intends to visit. A diffusion planner is far-sighted, synthesizing such a long-horizon trajectory, but it is only behavior-driven at execution, realizing each planned transition using only offline behavior.Each is strong exactly where the other is weak. We introduce DPDP to bring them together at the planner's execution step, where a planned transition becomes an action. Reading the planner's inverse-dynamics executor as a posterior over actions: the environment's transition likelihood relates an action to the planned next state, while the behavioral prior determines its weight under the offline data. Replacing this prior with a value-optimized diffusion policy preserves the planned transition likelihood and favors valuable actions that can serve the plan. We target the resulting clean-action distribution in two ways. A training-free sampler composes policy and inverse-dynamics scores at matched action noise levels and applies Langevin correction; this route uses a shared action noise schedule and an approximately common terminal prior. A preference objective instead amortizes the transfer into a single diffusion executor. Across D4RL, DPDP improves the planner it builds on, staying competitive with reactive diffusion policies on locomotion and leading on long-horizon navigation.