Driving · 2026-11-09
MDF++: Visuo-Tactile World Action Model with End-to-end Representation Learning
Zixuan Huang, Jinhao Yu, Zachary Charlick, Dmitry Berenson
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
MDF++ is a visuo-tactile world-action model that learns end-to-end, control-relevant latent representations using a latent diffusion backbone. It finetunes observation encoders jointly with a diffusion WAM via multi-task learning (policy learning, world-action modeling, forward/inverse dynamics, and random masking) and a selective gradient-flow rule that blocks gradients from each modality’s own denoising loss to prevent feature collapse. The authors evaluate on contact-rich manipulation tasks including real bolt screwing and nut mating.
Why it matters
The authors argue that existing world-action models often rely on frozen pretrained encoders optimized for reconstruction or semantics, which may yield representations misaligned with robot control—especially for tactile. MDF++ targets control-relevant representation learning for visuo-tactile manipulation by coupling end-to-end latent diffusion dynamics with cross-modal prediction and stabilizing training to avoid feature collapse.
Method
- End-to-end visuo-tactile WAM with a latent diffusion backbone that jointly trains modality encoders and a unified world-action model.
- Multi-task diffusion forcing: training on structured conditional distributions (policy learning, world-action modeling, forward dynamics, inverse dynamics, and random masking) to bias capacity toward control-relevant queries.
- Selective gradient-flow: each observation encoder is updated only by denoising losses from other streams (gradients stopped for its own modality loss) to avoid feature collapse.
Limitation
The paper text states that MDF++ comparisons against MDF [24] are imperfect because “Their Forge runs use end-of-trajectory padding, unlike the MDF++ reference, so the comparison does not fully isolate the objectives.”
Abstract (from arXiv)
Recently, world-action models (WAMs) have emerged as a powerful framework for robot manipulation. By jointly modeling action generation and future observation prediction, WAMs can act and anticipate the outcomes of the actions. However, existing WAMs typically rely on frozen pretrained encoders trained with reconstruction, which may not produce representations well suited for robot control. In this paper, we tackle the problem of learning physicallygrounded, control-relevant representations for WAMs. We propose MDF++, a visuo-tactile world-action model that learns latent representations of multisensory observations end-to-end with a diffusion backbone. While naive joint training can lead to feature collapse, MDF++ introduces a multi-task learning strategy that leverages cross-modal correlations to learn task-relevant representations. We evaluate MDF++ on a suite of contact-rich manipulation tasks, including realworld bolt screwing and nut mating, and show that it outperforms WAMs built on frozen pretrained representations by up to 50%. Additional videos are available on our website: jointmdf.github.io.
Related papers
- TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
- ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- Physical Canvas: A Vision Tactile World Model For Robot Policy Evaluation And Learning
- BAT-WAM: Bridging Action and Touch in World-Action Models for Dexterous Hands
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-09 · Indexed by Pub-AI: 2026-11-09 UTC
MDF++: Visuo-Tactile World Action Model with End-to-end Representation Learning
First announced by arXiv: 2026-11-09 · Original record ↗
- arxiv
- 2611.09293
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-09 · First detected: 2026-11-09 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.09293v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Recently, world-action models (WAMs) have emerged as a powerful framework for robot manipulation. By jointly modeling action generation and future observation prediction, WAMs can act and anticipate the outcomes of the actions. However, existing WAMs typically rely on frozen pretrained encoders trained with reconstruction, which may not produce representations well suited for robot control. In this paper, we tackle the problem of learning physicallygrounded, control-relevant representations for WAMs. We propose MDF++, a visuo-tactile world-action model that learns latent representations of multisensory observations end-to-end with a diffusion backbone. While naive joint training can lead to feature collapse, MDF++ introduces a multi-task learning strategy that leverages cross-modal correlations to learn task-relevant representations. We evaluate MDF++ on a suite of contact-rich manipulation tasks, including realworld bolt screwing and nut mating, and show that it outperforms WAMs built on frozen pretrained representations by up to 50%. Additional videos are available on our website: jointmdf.github.io.
