Driving · 2026-10-06
TAPDreamer: Transferable Adversarial Patches for World Action Models
Xuanyu Lu, Fengqing Jiang, Kaiyuan Zheng, Yichen Feng, Yaorui Ding, Yuetai Li, Zhen Xiang, Bhaskar Ramasubramanian, Basel Alomair, Luyao Niu, Radha Poovendran
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TAPDreamer is an untargeted test-time patch attack on world action models that uses only a public visual encoder and a few demo frames to optimize a fixed local pixel perturbation. The same frozen patch (about 6.5% of input) transfers across tasks and WAM architectures. In closed-loop control, it reduces FastWAM success from 97.65%→0.00% on LIBERO and 90.86%→0.00% on RoboTwin.
Why it matters
The authors show that even when defenses protect downstream action generation, shared visual encoders can remain vulnerable to persistent local perturbations that propagate into distributed latent shifts and cause closed-loop grasping/control failures. TAPDreamer requires no target-policy queries, uses only a public encoder for optimization, and transfers across tasks and models, highlighting the need for encoder-level robustness.
Method
- Optimize one fixed local pixel patch using a publicly released visual encoder: maximize mean L1 distance between clean and patched encoder outputs over all latent positions and channels across six source-task frames.
- Freeze patch pixels, area, and position, and deploy unchanged at every step; the attack requires no action labels, task rewards, simulator, victim weights, or policy outputs.
- Measure transfer in closed-loop task success on LIBERO and RoboTwin with FastWAM, and evaluate cross-model transfer to DreamWAM-uncond/joint and Motus.
Limitation
The attacker is assumed to have white-box access to a publicly released visual encoder (including weights and gradients) and a small set of public demonstration images from one source task.
Abstract (from arXiv)
World models learn to predict how their environment will evolve, making them an important foundation for general-purpose robotic control. Yet world action models depend on camera inputs whose manipulation can corrupt the visual representations used across tasks and action policies. Existing attacks on these models optimize against the victim's actions or predicted futures and therefore require access to target-model outputs. In this paper, we propose an attack, TAPDreamer, against world action models that instead uses a public encoder alone to construct a fixed local perturbation that transfers across tasks and action architectures. TAPDreamer requires no target-policy queries. Our key insight is that interactions between patch-induced changes in attention weights and value vectors broadcast a nearly identical representation shift far beyond the patch footprint, and this shift remains stable across task observations. Guided by this insight, TAPDreamer uses six frames from one source task to maximize the global L1 distance between clean and patched encoder representations. In closed-loop evaluation, one frozen patch per benchmark, covering about 6.5% of the input, reduces FastWAM's success rate from 97.7% to 0.0% across 40 LIBERO tasks and from 90.8% to 0.0% across 50 RoboTwin tasks; matched random patches retain 81.5% and 79.2% success. The same patches reduce success to 2.1% and 0.8% on two DreamWAM configurations and to 10.0% on Motus. These results show that protecting downstream action generation alone is insufficient: defenses for world action models must also secure shared visual encoders against persistent local perturbations.
Related papers
- PASA-KV: Attacking Action-Facing Visual Routes in World Action Models
- SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices
- GAWM: A Compact Latent World-Action Model for Multi-Task Robot Manipulation
- What Limits Cross-Task Generalization in World Action Models?
- Copper-Policy: Focus on the Representation for Robust Robot Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-05 · Indexed by Pub-AI: 2026-10-06 UTC
TAPDreamer: Transferable Adversarial Patches for World Action Models
Preprint posted: 2026-10-05 · Original record ↗
- arxiv
- 2610.06814
Explicit source versions (1)
- v3 · 2026-10-20
- Preprint available
Source date: 2026-10-05 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · atom:published (official metadata)
- Official evidence ↗ · atom:published (official metadata)
- Explicit source version detected · v3
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Official evidence ↗ · atom:id explicit version (official metadata)
Abstract · arxiv
World models learn to predict how their environment will evolve, making them an important foundation for general-purpose robotic control. Yet world action models depend on camera inputs whose manipulation can corrupt the visual representations used across tasks and action policies. Existing attacks on these models optimize against the victim's actions or predicted futures and therefore require access to target-model outputs. In this paper, we propose an attack, TAPDreamer, against world action models that instead uses a public encoder alone to construct a fixed local perturbation that transfers across tasks and action architectures. TAPDreamer requires no target-policy queries. Our key insight is that interactions between patch-induced changes in attention weights and value vectors broadcast a nearly identical representation shift far beyond the patch footprint, and this shift remains stable across task observations. Guided by this insight, TAPDreamer uses six frames from one source task to maximize the global L1 distance between clean and patched encoder representations. In closed-loop evaluation, one frozen patch per benchmark, covering about 6.5% of the input, reduces FastWAM's success rate from 97.7% to 0.0% across 40 LIBERO tasks and from 90.86% to 0.0% across 50 RoboTwin tasks; matched random patches retain 81.5% and 79.2% success. The same patches reduce success to 1.45% and 1.00% on two DreamWAM configurations and to 10.60% on Motus. These results show that protecting downstream action generation alone is insufficient: defenses for world action models must also secure shared visual encoders against persistent local perturbations.
