Driving · 2026-10-21
Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision
Minseong Kweon, Qiao Feng, Yiming Huang, Lingjie Liu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Touch-JEPA learns tactile representations from hand-object RGB video with limited tactile supervision. It uses geometry-guided self-supervised masked prediction to learn from video alone, then adds joint training where tactile contact and magnitude supervision is available only for a labeled subset. The authors report improved contact and tactile magnitude decoding from frozen features and transfer to robotic hands without robot tactile training.
Why it matters
The work aims to overcome the limited scale and coverage of paired tactile measurements by leveraging abundant hand-object video while still grounding representations in physical contact and tactile magnitude. It evaluates whether tactile quantities can be decoded from frozen representations and whether learned features transfer to robotic hands without robot tactile training.
Method
- Self-supervised pretraining: geometry-guided masked visual prediction predicts masked latent/visual features at anatomically corresponding hand-surface locations using spatial/temporal context, without tactile supervision.
- Joint representation learning: apply masked prediction to all videos while adding tactile supervision for contact and tactile magnitude on a labeled subset, grounding the shared encoder.
- Robotic transfer/control: for simulation planning, Touch-JEPA-AC freezes the vision encoder and learns an action-conditioned dynamics predictor using observed actions and robot state (no tactile targets for learning).
Abstract (from arXiv)
Understanding contact and tactile magnitude in hand-object interaction is a key step toward physically intelligent systems, yet learning such representations is limited by the cost and restricted coverage of tactile measurements. We present Touch-JEPA, a framework for learning tactile representations from hand-object videos with limited tactile supervision. We first introduce a self-supervised pretraining objective that learns from video alone without tactile supervision. Using video-derived hand geometry, Touch-JEPA predicts masked visual features at corresponding hand-surface locations from spatial and temporal context, encouraging the representation to capture latent contact dynamics. We then extend this formulation to joint representation learning, where masked prediction is applied to all videos while tactile supervision is available only for a labeled subset, grounding the shared representation in measured contact and magnitude. With self-supervised pretraining alone, simple supervised readouts on frozen Touch-JEPA features outperform supervised baselines in contact discrimination and tactile magnitude estimation across three human-hand benchmarks, and transfer to robotic hands without robot tactile training data. Joint learning with limited tactile supervision further improves contact and magnitude estimation on the human-hand benchmarks. Together, these results show that hand-object video can provide scalable self-supervision for tactile representation learning, while limited tactile measurements provide complementary physical grounding.
Related papers
- Tactile-Guided Robot Learning: From Representation Learning to Foundation Models
- ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
- TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
- Dexterous Tactile World Model
- V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC
Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28519
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Understanding contact and tactile magnitude in hand-object interaction is a key step toward physically intelligent systems, yet learning such representations is limited by the cost and restricted coverage of tactile measurements. We present Touch-JEPA, a framework for learning tactile representations from hand-object videos with limited tactile supervision. We first introduce a self-supervised pretraining objective that learns from video alone without tactile supervision. Using video-derived hand geometry, Touch-JEPA predicts masked visual features at corresponding hand-surface locations from spatial and temporal context, encouraging the representation to capture latent contact dynamics. We then extend this formulation to joint representation learning, where masked prediction is applied to all videos while tactile supervision is available only for a labeled subset, grounding the shared representation in measured contact and magnitude. With self-supervised pretraining alone, simple supervised readouts on frozen Touch-JEPA features outperform supervised baselines in contact discrimination and tactile magnitude estimation across three human-hand benchmarks, and transfer to robotic hands without robot tactile training data. Joint learning with limited tactile supervision further improves contact and magnitude estimation on the human-hand benchmarks. Together, these results show that hand-object video can provide scalable self-supervision for tactile representation learning, while limited tactile measurements provide complementary physical grounding.