Driving · 2026-11-10

BAT-WAM: Bridging Action and Touch in World-Action Models for Dexterous Hands

Dong Fang, Feixuan Huang, Yuanxin Zhong, Yuxiang Gao, Qi Song, Yunlong Wang, Xiaosong Jia, Yu-Gang Jiang

arXiv:2611.10967PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

BAT-WAM is a multimodal representation interface for world-action models that bridges tactile contact (Touch) and hand configuration (Joint) using RGB as continuous context. It learns separate codebooks/decoders for Touch and Joint and connects them with an RGB-conditioned continuous latent bridge for bidirectional learning. On HT-Bench, it reduces tactile reconstruction RMSE under full/partial masking versus an image-conditioned HandTouch checkpoint, and improves future tactile prediction versus OpenWAM with HandTouch. Real-robot experiments show higher success rates than OpenWAM baselines.

Why it matters

The authors address partially or fully missing joint-state and tactile modalities in dexterous manipulation with world action models. BAT-WAM explicitly learns conditional dependencies between Touch and Joint under RGB conditioning, supporting masked completion and conditional future tactile prediction within one interface. It reports improvements on HT-Bench tactile completion/prediction and higher real-robot task success and zero-shot generalization versus related OpenWAM variants.

Method

  • Separate tokenizers for Touch and Joint: Touch uses 196 spatial-grid tokens and Joint uses 8 tokens, each with its own encoder, projection, codebook, and physical decoder.
  • An RGB-conditioned continuous latent bridge: a shared Transformer maps Touch, Joint, and continuous RGB features into a common 256-dim space and models their relationships for bidirectional learning.
  • Two training regimes: static masked cross-modal reconstruction (complete missing modality) and multimodal temporal prediction for conditional future tactile prediction from historical Touch/Joint plus target-time RGB.

Limitation

Generalization to unseen sizes and shapes remains more challenging: the authors report BAT-WAM achieves only 37.5% and 27.5%, respectively, compared with 87.5% and 72.5% in the corresponding seen settings.

Abstract (from arXiv)

Dexterous manipulation with world action models (WAMs) relies on hand configuration and contact information, yet joint-state and whole-hand tactile observations may be partially or entirely missing. RGB images provide scene context and hand-object geometry but are insufficient to reliably recover these missing modalities. We further observe a correlation between hand state and tactile information. Motivated by this observation, we introduce BAT-WAM, a framework that bridges visual, joint-state, and tactile representations for WAMs. BAT-WAM learns separate codebooks and decoders for the tactile and action streams. An RGB-conditioned continuous latent bridge connects the two streams for bidirectional learning, while visual information serves as continuous context. Under the HTBench evaluation protocol, BAT-WAM reduces the root mean squared error (RMSE) of tactile reconstruction by 34.73% under fully masked conditions and 38.84% under partially masked conditions, relative to an image-conditioned HandTouch checkpoint. For future tactile prediction conditioned on target-time RGB and historical actions, it reduces mean RMSE by 85.56% relative to OpenWAM with HandTouch. Real-robot experiments further show that BAT-WAM consistently outperforms related WAM baselines in dexterous manipulation. Generalization experiments on unseen objects and scenes also demonstrate substantial improvements over baseline models. These findings support joint modeling of visual context, hand state, and touch as a representation approach for multimodal WAMs.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

BAT-WAM: Bridging Action and Touch in World-Action Models for Dexterous Hands

Dong Fang, Feixuan Huang, Yuanxin Zhong, Yuxiang Gao, Qi Song, Yunlong Wang, Xiaosong Jia, Yu-Gang Jiang

First announced by arXiv: 2026-11-10 · Original record ↗

arxiv
2611.10967
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.10967v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Dexterous manipulation with world action models (WAMs) relies on hand configuration and contact information, yet joint-state and whole-hand tactile observations may be partially or entirely missing. RGB images provide scene context and hand-object geometry but are insufficient to reliably recover these missing modalities. We further observe a correlation between hand state and tactile information. Motivated by this observation, we introduce BAT-WAM, a framework that bridges visual, joint-state, and tactile representations for WAMs. BAT-WAM learns separate codebooks and decoders for the tactile and action streams. An RGB-conditioned continuous latent bridge connects the two streams for bidirectional learning, while visual information serves as continuous context. Under the HTBench evaluation protocol, BAT-WAM reduces the root mean squared error (RMSE) of tactile reconstruction by 34.73% under fully masked conditions and 38.84% under partially masked conditions, relative to an image-conditioned HandTouch checkpoint. For future tactile prediction conditioned on target-time RGB and historical actions, it reduces mean RMSE by 85.56% relative to OpenWAM with HandTouch. Real-robot experiments further show that BAT-WAM consistently outperforms related WAM baselines in dexterous manipulation. Generalization experiments on unseen objects and scenes also demonstrate substantial improvements over baseline models. These findings support joint modeling of visual context, hand state, and touch as a representation approach for multimodal WAMs.