Driving · 2026-10-02
TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
Enyi Wang, Mingxin Wang, Quan Shi, Hetian Guo, Hongyu Wang, Xi Wang, Bin Qian, Yupeng Zheng, Wenxuan Song, Houde Liu, Yong Xu, Cheng Chi, Wenchao Ding, Yilun Chen, Yan Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TacDyn-WAM is a heterogeneous visuo-tactile world action model that predicts implicit tactile dynamics instead of reconstructing future tactile pixels. It learns TacRep via masked spatio-temporal tactile prediction with relational structure distillation. An ITDE predicts future tactile representations and their changes at multiple horizons in one forward pass, integrated with a visual expert via joint attention and a read-only tactile memory.
Why it matters
The authors argue that tactile prediction tied to pixel reconstruction becomes unreliable under deployment drift, since small contact changes can cause large pixel differences even if contact evolution remains predictable. By forecasting dynamics in a purpose-built tactile target space (TacRep) and predicting future changes, TacDyn-WAM aims for more robust tactile forecasting and better policy performance.
Method
- Predict implicit tactile dynamics in TacRep (a dynamics-aware tactile target space) rather than reconstructing future tactile observations.
- Train ITDE with masked spatio-temporal prediction and relational structure distillation to encode contact evolution while preserving local spatial relations.
- Use a one-pass ITDE that predicts future tactile representations and their deltas at multiple horizons, integrated with a visual expert via joint attention and a read-only tactile memory.
Abstract (from arXiv)
World action models improve robotic manipulation by conditioning actions on predicted futures, yet existing tactile variants largely inherit video-generation pipelines that reconstruct future tactile observations through iterative denoising. Such prediction can become unreliable under deployment drift: small changes in contact position or force may substantially alter tactile pixels even when the underlying contact evolution remains predictable. We introduce TacDyn-WAM, a heterogeneous visuo-tactile world action model that predicts implicit tactile dynamics rather than reconstructing future tactile observations. It learns TacRep, a dynamics-aware tactile target space trained through masked spatio-temporal prediction on tactile clips and regularized by relational structure distillation. A visual expert and an Implicit Tactile Dynamics Expert predict future visual and tactile representations in separate target spaces while interacting through joint attention; the tactile expert predicts future representations and their changes at multiple horizons in a single forward pass, and a read-only tactile memory supplies the current tactile state. On UniVTAC, TacDyn-WAM achieves an average success rate of 81.5% using only the provided demonstrations, reaching state-of-the-art-level performance and remaining competitive with models pretrained on large-scale visuo-tactile trajectories. Ablations confirm the benefits of both tactile pathways and TacRep over pixel-reconstruction and static alternatives. On five real-robot tasks, TacDyn-WAM reaches 71.0% average success, and modest-scale pretraining raises it to 85.0%, further validating our method.
Related papers
- ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
- Tactile-Guided Robot Learning: From Representation Learning to Foundation Models
- Physical Canvas: A Vision Tactile World Model For Robot Policy Evaluation And Learning
- Dexterous Tactile World Model
- Interaction Latents as Actions for Contact-Rich Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC
TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2610.00638
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World action models improve robotic manipulation by conditioning actions on predicted futures, yet existing tactile variants largely inherit video-generation pipelines that reconstruct future tactile observations through iterative denoising. Such prediction can become unreliable under deployment drift: small changes in contact position or force may substantially alter tactile pixels even when the underlying contact evolution remains predictable. We introduce TacDyn-WAM, a heterogeneous visuo-tactile world action model that predicts implicit tactile dynamics rather than reconstructing future tactile observations. It learns TacRep, a dynamics-aware tactile target space trained through masked spatio-temporal prediction on tactile clips and regularized by relational structure distillation. A visual expert and an Implicit Tactile Dynamics Expert predict future visual and tactile representations in separate target spaces while interacting through joint attention; the tactile expert predicts future representations and their changes at multiple horizons in a single forward pass, and a read-only tactile memory supplies the current tactile state. On UniVTAC, TacDyn-WAM achieves an average success rate of 81.5% using only the provided demonstrations, reaching state-of-the-art-level performance and remaining competitive with models pretrained on large-scale visuo-tactile trajectories. Ablations confirm the benefits of both tactile pathways and TacRep over pixel-reconstruction and static alternatives. On five real-robot tasks, TacDyn-WAM reaches 71.0% average success, and modest-scale pretraining raises it to 85.0%, further validating our method.