Driving · 2026-10-14

DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving

Shuzheng Wang, Haoxuan Li, Shengzhen Li, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DriveWLA is a unified World-Language-Action model for autonomous driving that uses three MoT “experts” (World for future prediction, Language for semantic understanding, Action for trajectory generation) with Cross-Expert Attention to control information flow and align representations. The Action Expert is refined with Flow-GRPO reinforcement learning post-training beyond imitation learning. The authors report strong open/closed-loop performance and zero-shot generalization across driving simulators and datasets.

Why it matters

Trajectory generation needs both semantic scene understanding and future dynamics. The authors address integration challenges by jointly training specialized experts with Cross-Expert Attention under trajectory supervision, then improving planning beyond imitation with Flow-GRPO. This targets better open-loop/closed-loop driving performance and zero-shot transfer for autonomous driving.

Method

Three-expert MoT architecture: World Expert for future dynamics (video-generation pretrained), Language Expert for semantic understanding (frozen driving-pretrained VLM + VLM Adapter), and Action Expert for trajectory generation (DiT with flow-matching policy).
Cross-Expert Attention controls information exchange across experts via a masked attention visibility scheme (World↔Language bidirectional; Action attends to World/Language; World/Language do not receive Action outputs).
Imitation learning jointly optimizes experts with a flow-matching behavior cloning objective; post-training refines the Action Expert using Flow-GRPO with factorized advantage estimation and variance-based shrinkage.

Abstract (from arXiv)

Vision-Language Models (VLMs) and world models have demonstrated strong capabilities in semantic understanding and future scene evolution for autonomous driving. However, effectively integrating both capabilities for trajectory generation remains challenging. In this work, we propose DriveWLA, a unified World-Language-Action model based on Mixture-of-Transformers (MoT) that jointly models semantic understanding, future scene dynamics, and trajectory generation. DriveWLA consists of three experts: a World Expert for future prediction, a Language Expert for semantic understanding, and an Action Expert for trajectory generation. We further introduce Cross-Expert Attention, which enables controlled information flow across experts while aligning semantic and dynamic representations. To improve trajectory generation beyond imitation learning, we apply Flow-GRPO reinforcement learning post-training to refine the Action Expert. Extensive experiments show that DriveWLA achieves superior performance in both open-loop and closed-loop evaluation. DriveWLA achieves 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, demonstrating strong performance among VLM-based and world-model-based planners. Without specific fine-tuning, DriveWLA further demonstrates strong zero-shot generalization to HUGSIM, nuScenes, and Bench2Drive. These results validate the effectiveness of our unified World-Language-Action modeling, establishing DriveWLA as a generalizable framework for autonomous driving.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-14 UTC

preprint