Pub-AI: AI in Science

Robotics · 2026-09-30

DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction

Wenliang Guo, Zhanbo Huang, Yu Kong

arXiv:2609.36454PDF

Auto-summarized: this summary was generated by a language model from the paper’s text and has not been reviewed by an editor. Check the paper before relying on it. Benchmark numbers appear only after a human has verified them.

TL;DR

DynamicHOI reconstructs hand-object interaction (HOI) trajectories from monocular RGB video using geometry-grounded diffusion refinement plus coupled hand-object physics. It derives hand generalized forces via articulated inverse dynamics and object wrenches via Newton–Euler dynamics, couples them through contact-force transfer, and recovers active hand actuation regularized by a probabilistic prior to suppress mechanically implausible motion.

Why it matters

The authors argue that trajectory-level supervision can yield visually plausible but mechanically inconsistent motion. DynamicHOI adds dynamics-level supervision by aligning hand generalized forces and object wrenches between reconstructed and ground-truth trajectories, and couples the two through contact-force transfer with an actuation prior. The paper further evaluates downstream effects for robot manipulation learning and egocentric hand world-model generation.

Method

  • Geometry-grounded diffusion refinement: encode noisy hand/object trajectories in mesh space, extract DINOv2 features, project 3D geometric anchors from initial estimates to retrieve visual evidence, and iteratively denoise with interaction-aware attention.
  • Dynamics-level supervision: compute hand generalized forces from reconstructed trajectories via articulated inverse dynamics and compute rigid object wrenches via Newton–Euler dynamics; align these mechanical quantities between reconstructed and ground-truth trajectories.
  • Coupled hand-object dynamics: infer contact forces to explain object wrenches, transfer them to hand generalized coordinates to recover active hand actuation, and regularize actuation with a probabilistic prior to penalize unlikely actuation.

Limitation

The authors attribute limited improvement from ℒ_act partly to large variability of video content and errors in the forces estimated by the Euler-Lagrange and Newton-Euler equations, which may propagate into the inferred actuation distribution.

Abstract (from arXiv)

We study hand-object interaction (HOI) reconstruction from monocular RGB videos, where partial observations can produce visually plausible yet mechanically inconsistent trajectories. Existing methods mainly enforce visual and geometric agreement, leaving the underlying interaction dynamics insufficiently constrained. We propose DynamicHOI, a physics-aware HOI reconstruction framework combining geometry-grounded diffusion refinement with coupled hand-object dynamics. Geometry spatially grounds visual evidence for trajectory refinement, while articulated inverse dynamics and Newton-Euler dynamics derive hand generalized forces and object wrenches for dynamics-level supervision. We further couple hand and object dynamics through contact-force transfer and recover active hand actuation as an interaction-level physical quantity. We formulate its empirical magnitude distribution into a probabilistic prior that penalizes unlikely actuation and suppresses mechanically implausible reconstructed motion. Experiments on three HOI datasets show consistent improvements in both hand and object reconstruction. The reconstructed trajectories further benefit downstream applications including hand world-model generation and robotic manipulation learning, demonstrating the value of physics-aware HOI modeling beyond reconstruction.

Related papers