Driving · 2026-11-13

Towards Robust Vision-Language Reasoning: A Multi-Stage Optimization Pipeline with Masked Diffusion Model

Shunming Liu, Pei Huang, Yajing Sun, Huaxiao Liu

arXiv:2611.15238PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose a multi-stage optimization pipeline for discrete diffusion vision-language models in autonomous driving, covering (1) trajectory discretization in supervised fine-tuning to suppress hallucinations, (2) metric-guided diversity filtering during reinforcement learning to improve exploration, and (3) Dynamic Autonomous Driving Decoding (DAD) at inference to adapt decoding order by scene complexity. They develop Masked Diffusion Drive (MD-Drive).

Why it matters

The paper targets domain gaps when applying discrete diffusion generation to driving, including noise from text serialization, RL learning efficiency/convergence bottlenecks, and inference mismatch with driving logic. By optimizing planning representations, filtering RL starting points using accuracy/diversity metrics, and adapting inference order using confidence trends, the approach aims to improve driving performance while balancing computational cost.

Method

  • SFT: trajectory discretization maps continuous planning attributes (e.g., 2D coordinates, heading angle) to 1D codebook tokens, replacing planning components with special tokens for supervised learning.
  • RL: metric-guided diversity filtering screens SFT checkpoints using pass@1 for accuracy and pass@n for exploration diversity, then initializes GRPO from the selected policy; refined RL dataset selects mid/hard based on rollout difficulty.
  • Inference (DAD): computes per-token confidence from logits and accumulates confidence over proximal vs distal trajectory segments to select Near-to-Far or Far-to-Near decoding order, avoiding frequent switching across generation direction.
Abstract (from arXiv)

Discrete diffusion generation is emerging as a competitive alternative to autoregressive generation by denoising the entire token sequence through iterative refinement. Its global planning and progressive correction capabilities are particularly well suited to autonomous driving. However, transferring this generation paradigm from general-purpose settings to driving introduces substantial domain gaps in both training and inference. To unlock its potential, we propose a multi-stage optimization pipeline for standard vision-language model architectures, without auxiliary action experts or specialized mixture-of-experts Transformer, spanning post-training and inference. During supervised fine-tuning, we optimize the representation of planning inputs to suppress hallucinations in task responses. During reinforcement learning, we introduce metric-guided diversity filtering to identify stronger learning starting points and improve policy exploration. At inference time, we propose Dynamic Autonomous Driving Decoding (DAD), which adaptively adjusts decoding modes according to scene complexity. Based on this pipeline, we develop Masked Diffusion Drive (MD-Drive), improving its performance by 29.7\% over the unoptimized model. Experiments on the NAVSIM closed loop benchmark and the nuScenes and DriveCoT open loop datasets show that MD-Drive outperforms autoregressive VLM baselines with the same architecture, while achieving a more favorable balance between performance and computational cost than systems with different architectures through discrete diffusion training and inference.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

Towards Robust Vision-Language Reasoning: A Multi-Stage Optimization Pipeline with Masked Diffusion Model

Shunming Liu, Pei Huang, Yajing Sun, Huaxiao Liu

First announced by arXiv: 2026-11-13 · Original record ↗

arxiv
2611.15238
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.15238v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Discrete diffusion generation is emerging as a competitive alternative to autoregressive generation by denoising the entire token sequence through iterative refinement. Its global planning and progressive correction capabilities are particularly well suited to autonomous driving. However, transferring this generation paradigm from general-purpose settings to driving introduces substantial domain gaps in both training and inference. To unlock its potential, we propose a multi-stage optimization pipeline for standard vision-language model architectures, without auxiliary action experts or specialized mixture-of-experts Transformer, spanning post-training and inference. During supervised fine-tuning, we optimize the representation of planning inputs to suppress hallucinations in task responses. During reinforcement learning, we introduce metric-guided diversity filtering to identify stronger learning starting points and improve policy exploration. At inference time, we propose Dynamic Autonomous Driving Decoding (DAD), which adaptively adjusts decoding modes according to scene complexity. Based on this pipeline, we develop Masked Diffusion Drive (MD-Drive), improving its performance by 29.7\% over the unoptimized model. Experiments on the NAVSIM closed loop benchmark and the nuScenes and DriveCoT open loop datasets show that MD-Drive outperforms autoregressive VLM baselines with the same architecture, while achieving a more favorable balance between performance and computational cost than systems with different architectures through discrete diffusion training and inference.