Driving · 2026-10-02
CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight
Chensheng Peng, Wenhao Ding, Ran Tian, Zewei Zhou, Jef Packer, Maximilian Igl, Peter Karkus, Yan Wang, Masayoshi Tomizuka, Boris Ivanovic, Marco Pavone, Yuxiao Chen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
CtrlWAM trains world action models with physically aligned denoising: instead of pairing noised actions with a video still tied to the recorded future, it perturbs actions in a simulator and pairs them with their rendered visual consequences (then corrupts video with fresh noise). It also introduces warped video–action noise schedules and a structured multi-agent action interface for driving and bimanual manipulation.
Why it matters
The authors argue that standard joint noising introduces intent–foresight mismatch during denoising, since perturbed actions imply counterfactual futures while the noised video remains tied to the GT recording. CtrlWAM uses aligned noising to supply physically corresponding action–video pairs, and evaluates improvements in command following, video–action consistency, and manipulation motion fidelity.
Method
- Physically aligned noising: simulate perturbed action commands, render their consequences, and train the model to recover the recorded clean actions and recorded future video at the corresponding timestamps.
- Warped video–action noise schedules: assign higher video noise than action noise at the same action noise level to keep visual layout responsive while action estimates evolve, then refine details later.
- Structured multi-agent conditioning: extend the action interface from ego-only to an ego block plus K selected agent streams, enabling joint multi-agent prediction or conditioning video on supplied agent commands.
Limitation
The aligned noising pipeline requires a simulator to execute perturbed actions and render their consequences. The multi-agent extension is limited to driving because the authors lack suitable dynamic robotics datasets with actor-level action–observation pairs.
Abstract (from arXiv)
World action models (WAMs) jointly predict actions (intent) and visual future (foresight). Standard training adds noise to recorded actions and video simultaneously, but such training paradigms introduce a mismatch: perturbed actions imply counterfactual future visual, while the noised video remains tied to the GT recording. In low-noise regime, the scene geometry and even the dynamic behavior remain clearly visible from the noisy future frames despite the added noise. We present CtrlWAM, which executes perturbed actions in a simulator and pairs them with their noised visual consequences for joint WAM learning. To accommodate the different denoising requirements of video and actions, we introduce warped video--action noise schedules that aim to keep visual layout responsive as action predictions evolve. We further extend the action interface from ego-only control to a variable number of agent streams, allowing a unified model to represent predicted or commanded futures for multiple agents. Driving experiments show more accurate action forecasts, closer agreement between generated video and actions, and better following of supplied commands; robotics experiments show stronger motion fidelity and controllability. Matched controls support the benefit of off-path renders for command following and manipulation fidelity. Together, these findings contribute to a more controllable world action model. Project page: https://ctrl-wam.github.io/
Related papers
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
- De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
- RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations
- World Action Modeling with Progressive Visual Planning
- Achieve What You Imagined: Learning to Align Actions with Visual Plans
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC
CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight
Preprint posted: 2026-10-01 · Original record ↗
- arxiv
- 2610.00859
Explicit source versions (1)
- v1 · 2026-10-01
- Preprint available
Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World action models (WAMs) jointly predict actions (intent) and visual future (foresight). Standard training adds noise to recorded actions and video simultaneously, but such training paradigms introduce a mismatch: perturbed actions imply counterfactual future visual, while the noised video remains tied to the GT recording. In low-noise regime, the scene geometry and even the dynamic behavior remain clearly visible from the noisy future frames despite the added noise. We present CtrlWAM, which executes perturbed actions in a simulator and pairs them with their noised visual consequences for joint WAM learning. To accommodate the different denoising requirements of video and actions, we introduce warped video--action noise schedules that aim to keep visual layout responsive as action predictions evolve. We further extend the action interface from ego-only control to a variable number of agent streams, allowing a unified model to represent predicted or commanded futures for multiple agents. Driving experiments show more accurate action forecasts, closer agreement between generated video and actions, and better following of supplied commands; robotics experiments show stronger motion fidelity and controllability. Matched controls support the benefit of off-path renders for command following and manipulation fidelity. Together, these findings contribute to a more controllable world action model. Project page: https://ctrl-wam.github.io/