Driving · 2026-10-02
DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
Youngjun Jun, Kyumin Choi, Youngmin Kim, Seonghyun Jin, Sunwoo Park, Jangho Park, Jong Chul Ye
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DriftOPD is a teacher-free, rollout-free framework for sequence-level on-policy distillation of continuous VLA action experts into one-step policies. It decomposes sequence-level reverse KL into a chunk-level reverse-KL term and a future-potential term. DriftOPD optimizes the local term with a one-step drifting objective and approximates the future term with a Q-function critic learned from offline demonstrations, using only offline data and one-step action generation.
Why it matters
The authors target a common gap in chunk-level one-step VLA training: optimizing local action likelihood without explicitly capturing long-horizon task success. DriftOPD aims to incorporate long-horizon effects into one-step action experts without costly closed-loop rollouts or a separate teacher, and reports general performance improvements across multiple VLA architectures in simulation and real-world manipulation.
Method
- Formulate sequence-level reverse-KL over action sequences in VLA as a sum of a chunk-level reverse-KL term and a future term capturing long-horizon effects.
- Make the local reverse-KL term tractable for continuous action experts via a one-step distribution-matching drifting objective using a drifting field.
- Approximate the future-potential term with a frozen action-value critic Qϕ trained from offline demonstrations, enabling rollout-free one-step distillation.
Limitation
The authors’ rollout-free DriftOPD uses a critic trained only from demonstrations; its action-gradient quality is limited by demonstration coverage. This limitation is particularly evident when the one-step teacher already matches its multi-step counterpart (e.g., several LIBERO and LIBERO-Plus settings), where the current critic provides little additional improvement beyond the teacher/student baseline.
Abstract (from arXiv)
Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control. While chunk-level training is convenient across robot embodiments, it optimizes local action likelihood without explicitly accounting for long-horizon task success. Sequence-level reinforcement learning can address this limitation, but typically requires policy rollouts and closed-loop interaction, which are costly for real-robot manipulation. We introduce DriftOPD, a teacher-free, rollout-free framework for sequence-level on-policy distillation of continuous VLA action experts. We show that the sequence-level reverse Kullback-Leibler (KL) divergence decomposes into a chunk-level reverse-KL term and a future-potential term that captures the long-horizon effect of the current action. DriftOPD optimizes these two terms using a one-step drifting objective and a Q-function critic learned from offline demonstrations, respectively, enabling sequence-level optimization with only offline data and one-step action generation. Across multiple VLA architectures in simulation and real-world manipulation, DriftOPD generally outperforms existing one-step distillation baselines while achieving task success performance comparable to multi-step teacher policies. These results demonstrate that long-horizon behavior can be effectively distilled into one-step VLA action experts without online interaction or a separate teacher.
Related papers
- FastOPD: On-Policy Distillation for Lightweight VLA Deployment
- Fast-VLA: You Only Think Once for Robot Manipulation
- WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
- PaceVLA: Adaptive Temporal Action Quantization in Vision-Language-Action Models
- WoVR+: World Model as Reliable Simulator for Versatile VLA Policy Optimization with RL
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-29 · Indexed by Pub-AI: 2026-10-02 UTC
DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
Preprint posted: 2026-09-29 · Original record ↗
- arxiv
- 2610.00317
Explicit source versions (1)
- v1 · 2026-09-29
- Preprint available
Source date: 2026-09-29 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control. While chunk-level training is convenient across robot embodiments, it optimizes local action likelihood without explicitly accounting for long-horizon task success. Sequence-level reinforcement learning can address this limitation, but typically requires policy rollouts and closed-loop interaction, which are costly for real-robot manipulation. We introduce DriftOPD, a teacher-free, rollout-free framework for sequence-level on-policy distillation of continuous VLA action experts. We show that the sequence-level reverse Kullback-Leibler (KL) divergence decomposes into a chunk-level reverse-KL term and a future-potential term that captures the long-horizon effect of the current action. DriftOPD optimizes these two terms using a one-step drifting objective and a Q-function critic learned from offline demonstrations, respectively, enabling sequence-level optimization with only offline data and one-step action generation. Across multiple VLA architectures in simulation and real-world manipulation, DriftOPD generally outperforms existing one-step distillation baselines while achieving task success performance comparable to multi-step teacher policies. These results demonstrate that long-horizon behavior can be effectively distilled into one-step VLA action experts without online interaction or a separate teacher.