Driving · 2026-11-06

WAM-Sim0.1: World Anchored Memory Meets Online RL for Consistent and Realistic Driving Simulation

Jiahao Cui, Jiaye Li, Wenhao Zhang, Yaoyi Li, Shan Luan, Quanhui Tang, Jianhua Han, Lin Cheng, Hang Xu, Siyu Zhu

arXiv:2611.07939PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

WAM-Sim is a driving world model that uses a persistent hierarchical world anchored memory (structure, scene, and write-once instance tokens) to reduce structural drift and identity drift in long rollouts. It applies an online post-training method, Drive DiffusionNFT, using flow-matching RL with hierarchy-aligned visual rewards and a physical judge to restore visual sharpness and physical plausibility.

Why it matters

The authors target long-horizon driving simulation where autoregressive video world models degrade via structural drift, identity drift, and quality decay. WAM-Sim’s persistent, multi-scale memory is intended to anchor road geometry, geolocated background appearance, and traffic participant identity, while online flow-matching RL is used to enforce physical plausibility and fine-grained visual fidelity during self-generated rollouts.

Method

  • World anchored memory with three persistent tiers: structural HD-map tokens, read-only geolocated scene tokens, and write-once instance tokens cached by track identifiers and retrieved via instance-masked cross-attention.
  • Hierarchical memory injection: structural memory provides additive trajectory-aligned geometry; scene memory is routed per temporal segment; instance memory conditions only foreground regions matching tracked instances.
  • Drive DiffusionNFT: online post-training that optimizes self-generated rollouts via flow-matching reinforcement learning using hierarchy-aligned rewards (scene, object, physics) guided by a vision-language physical judge.

Limitation

Memory retrieval latency and attention memory footprint scale linearly with the number of tracked foreground instances, introducing computational overhead in hyper-dense traffic scenes containing over 100 concurrent agents. The write-once instance appearance bank assumes rigid visual properties, constraining extreme non-rigid state transitions (e.g., severe vehicle damage or drastic open/close deformations). Background scene retrieval depends on geolocated reference availability and may degrade to ungrounded autoregressive generation in unmapped or highly dynamic construction zones.

Abstract (from arXiv)

Autoregressive driving world models systematically suffer from structural drift, identity drift, and quality decay over long rollout horizons due to transient conditioning signals that lack persistent state. We present WAM-Sim, a driving world model anchored by a persistent hierarchical memory and optimized via online flow-matching RL. WAM-Sim structures persistent context across three temporal and semantic scales: pose-aligned structural tokens encoding persistent road topology, read-only scene tokens supplying geolocated background appearance, and write-once instance tokens preserving individual traffic participant identities across occlusions and chunk boundaries. To restore visual sharpness and physical plausibility degraded by multi-scale conditioning, we introduce an online post-training framework that optimizes self-generated rollouts via flow-matching RL guided by hierarchy-aligned visual rewards and a vision-language model physical judge. The invariant memory empirically confines policy optimization within the anchored context, helping suppress semantic collapse and reward hacking, while online RL systematically enforces physical commonsense and fine-grained visual fidelity. On the nuScenes benchmark, WAM-Sim achieves a superior generation fidelity (FID=3.47, FVD=25.72) and strong downstream perception performance (24.16 mAP, 71.23 road mIoU). On minute-scale rollouts of self-collected data, WAM-Sim substantially mitigates structural and identity drift, preserving high scene similarity and subject identity coherence across extended horizons. Project page: https://fudan-generative-vision.github.io/WAM-Sim.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-06 · Indexed by Pub-AI: 2026-11-06 UTC

preprint

WAM-Sim0.1: World Anchored Memory Meets Online RL for Consistent and Realistic Driving Simulation

Jiahao Cui, Jiaye Li, Wenhao Zhang, Yaoyi Li, Shan Luan, Quanhui Tang, Jianhua Han, Lin Cheng, Hang Xu, Siyu Zhu

First announced by arXiv: 2026-11-06 · Original record ↗

arxiv
2611.07939
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-06 · First detected: 2026-11-06 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.07939v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Autoregressive driving world models systematically suffer from structural drift, identity drift, and quality decay over long rollout horizons due to transient conditioning signals that lack persistent state. We present WAM-Sim, a driving world model anchored by a persistent hierarchical memory and optimized via online flow-matching RL. WAM-Sim structures persistent context across three temporal and semantic scales: pose-aligned structural tokens encoding persistent road topology, read-only scene tokens supplying geolocated background appearance, and write-once instance tokens preserving individual traffic participant identities across occlusions and chunk boundaries. To restore visual sharpness and physical plausibility degraded by multi-scale conditioning, we introduce an online post-training framework that optimizes self-generated rollouts via flow-matching RL guided by hierarchy-aligned visual rewards and a vision-language model physical judge. The invariant memory empirically confines policy optimization within the anchored context, helping suppress semantic collapse and reward hacking, while online RL systematically enforces physical commonsense and fine-grained visual fidelity. On the nuScenes benchmark, WAM-Sim achieves a superior generation fidelity (FID=3.47, FVD=25.72) and strong downstream perception performance (24.16 mAP, 71.23 road mIoU). On minute-scale rollouts of self-collected data, WAM-Sim substantially mitigates structural and identity drift, preserving high scene similarity and subject identity coherence across extended horizons. Project page: https://fudan-generative-vision.github.io/WAM-Sim.