Robotics · 2026-10-20
ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
Yuxiang Ma, Weihan Xu, Pengfei Ye, Anna Mei, Yuxin Ray Song, Tingle Li, Haonan Chen, Edward H. Adelson
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ACTS is an action-conditioned visuo-tactile world model trained only on human physical interaction. Operators manipulate handheld GelSight sensors while the system records multi-view video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, forming React (4.9 hours). ACTS jointly predicts future scene, wrist, and tactile observations and is evaluated zero-shot on robot teleoperation.
Why it matters
Contact-rich manipulation needs data capturing contact formation, sliding, and release under varying force and geometry. ACTS uses a sensor-level action representation (motion plus force-informed displacement) so humans and robots share the same action space, enabling learning from human interactions and transferring to robots without fine-tuning.
Method
- Human data: React collected by five operators manipulating handheld GelSight sensors; records top-down scene video, wrist cameras, GelSight tactile images, 6-DoF poses, and tactile-derived normal force (tracked at 30 Hz).
- Sensor-level actions: force-aware sensor motion augments relative sensor translation with a normal displacement driven by tactile-estimated normal force change; actions are also encoded in a view-aligned scene geometry representation.
- Model: ACTS is trained to jointly predict future scene, wrist, and tactile observations from observation histories and the full action window, using frozen pretrained observation encoders and an action-conditioned generator initialized from Cosmos3.
Limitation
Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard; wrist views can blur or hallucinate structure.
Abstract (from arXiv)
Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.
Related papers
- TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
- Dexterous Tactile World Model
- Physical Canvas: A Vision Tactile World Model For Robot Policy Evaluation And Learning
- Interaction Latents as Actions for Contact-Rich Manipulation
- Tactile-Guided Robot Learning: From Representation Learning to Foundation Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-16 · Indexed by Pub-AI: 2026-10-20 UTC
ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
Preprint posted: 2026-10-16 · Original record ↗
- arxiv
- 2610.24285
Explicit source versions (1)
- v1 · 2026-10-16
- Preprint available
Source date: 2026-10-16 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.