JEPA · 2026-10-20

V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos

Ruicheng Li, Qixiu Li, Yu Deng, Sicheng Xu, Jiaolong Yang, Baining Guo

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

V-JEPA Hand is a single feed-forward video model that jointly reconstructs bilateral 3D hand motion and camera egomotion from egocentric clips in one forward pass. Using a self-supervised V-JEPA 2.1 encoder plus a query-based predictor, it outputs per-frame hand poses/presence and camera motion, expressing hand trajectories in a common world frame. The model accepts known focal lengths or predicts them and is trained end to end on mixed pseudo-labelled and high-precision 3D annotations.

Why it matters

The authors target metric 3D hand trajectories in a consistent world frame from egocentric video without a multi-stage pipeline (e.g., separate hand reconstruction and SLAM). They report lower world-frame trajectory errors than compared SLAM-based baselines and evaluate on ARCTIC, HOT3D, and EgoStandard, with ablations supporting benefits of video pretraining, temporal context, and mixed supervision.

Method

Joint, single-stage model: a V-JEPA 2.1 video encoder extracts spatio-temporal features and a query-based ViT predicts per-frame bilateral MANO pose/presence plus camera motion from the same representation.
World-frame trajectories: predicted camera egomotion is used to express hand trajectories in a common world frame; focal lengths are accepted when known or predicted via an intrinsics token.
Training: end-to-end single-stage optimization on a mixture of pseudo-labelled Ego4D/in-house videos (via VITRA) and high-quality 3D annotations from ARCTIC and HOT3D, with losses for pose, 3D joints, reprojection, jitter, presence, camera motion, and (withheld) focal lengths.

Limitation

The authors state that single-frame world errors in their ablations are computed using ground-truth camera rotation and translation (oracle), and are excluded from world-column ranking.

Abstract (from arXiv)

Egocentric video supports diverse applications, including robot learning, augmented reality, and human activity understanding. Recovering metric 3D hand trajectories from such video requires estimating both hand motion and camera egomotion to express the hands in a consistent reference frame. Many existing pipelines recover hand and camera motion in separate stages, combining hand detection, hand reconstruction, camera tracking, and trajectory refinement. We present V-JEPA Hand, a simple feed-forward video model that jointly reconstructs bilateral hand motion and camera motion from egocentric clips. It uses a self-supervised V-JEPA 2.1 encoder to extract spatio-temporal features from full-frame video and a query-based predictor to decode these features into per-frame hand poses, hand presence, and camera motion. We use the predicted camera motion to express the hand trajectories in a common world frame. The model accepts known focal lengths or predicts them when unavailable. It reconstructs each clip in a single forward pass. We train both components end to end in a single stage on a mixture of diverse pseudo-labelled videos and high-precision 3D annotations. A single model achieves the lowest coverage-aware mean per-joint position error (MPJPE) among comparable baselines on ARCTIC, HOT3D, and EgoStandard. It also yields lower world-frame trajectory errors than the compared SLAM-based pipelines. Ablations support the benefits of video pretraining, temporal context, and mixed supervision. V-JEPA Hand achieves a forward-pass throughput of 93.4 FPS on a single RTX A6000. Project page: https://qft-333.github.io/V-JEPA-Hand-Page/.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-20 UTC

preprint