Video World Models · 2026-10-12
Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
Peiyan Tu, Cong Wang, Hanxin Zhu, Xin Jin, Zhibo Chen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
HF-WM is an action-conditioned world model that learns interaction dynamics from egocentric human videos and improves failure modeling via GRPO post-training on counterfactual action perturbations. It uses a dual-branch architecture with stabilized pointflow supervision to bridge the human–robot embodiment gap, and GRPO with action alignment plus physical plausibility rewards to model plausible failure outcomes without paired failure video supervision.
Why it matters
The authors address two gaps in action-conditioned world modeling: limited coverage of diverse interaction dynamics (especially across embodiments) and underrepresentation of failure dynamics. HF-WM aims to better predict action consequences by learning from human interaction data with stabilized motion representations and by learning failure behavior using reward-guided optimization rather than relying on paired failure videos.
Method
- Stage 1: train an action-conditioned video world model with stabilized pointflow supervision in a dual-branch setup (video generation + auxiliary pointflow head) to learn interaction motion while suppressing camera-induced motion.
- Bridge embodiments: map human hand and robot arm actions into a shared 2D skeleton spatial action space and supervise pointflow using camera-motion-compensated tracked point trajectories.
- Stage 2: perform GRPO post-training using perturbed actions from successful demonstrations; use an action-alignment reward (pairwise action following scorer) and a physical plausibility reward (tiered checklist evaluation) to learn failure dynamics without paired failure-video supervision.
Abstract (from arXiv)
Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions. However, existing robotic datasets provide limited coverage of the diverse interaction dynamics required for generalizable world modeling. Large-scale egocentric human videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning. Meanwhile, the predominance of successful demonstrations leaves failure dynamics substantially underrepresented, limiting the ability of world models to anticipate unsuccessful executions. To address these limitations, we introduce HF-WM, an action-conditioned world model that combines Human interaction learning with Failure-aware post-training. To bridge the embodiment gap, we develop a dual-branch framework that couples video generation with auxiliary pointflow prediction, using stabilized pointflow as a unified motion representation across human and robot interactions while suppressing egocentric camera motion. To improve failure modeling, we further perform Group Relative Policy Optimization (GRPO) post-training on perturbed actions derived from successful demonstrations, guided by action alignment and physical plausibility rewards, enabling the model to learn plausible failure dynamics without paired ground-truth videos. Extensive experiments demonstrate that HF-WM achieves state-of-the-art performance in action-conditioned video generation and exhibits strong utility for policy evaluation in downstream robotic tasks.
Related papers
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
- World Action Learning via Interaction-Centric Spectral Latent Guidance
- Learning Transferable World-Action Models from Task-Paired Human Videos
- DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training
- OpenWAM: An Open Framework for Composable World-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-12 UTC
Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
Preprint posted: 2026-10-09 · Original record ↗
- arxiv
- 2610.13522
Explicit source versions (1)
- v1 · 2026-10-09
- Preprint available
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions. However, existing robotic datasets provide limited coverage of the diverse interaction dynamics required for generalizable world modeling. Large-scale egocentric human videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning. Meanwhile, the predominance of successful demonstrations leaves failure dynamics substantially underrepresented, limiting the ability of world models to anticipate unsuccessful executions. To address these limitations, we introduce HF-WM, an action-conditioned world model that combines Human interaction learning with Failure-aware post-training. To bridge the embodiment gap, we develop a dual-branch framework that couples video generation with auxiliary pointflow prediction, using stabilized pointflow as a unified motion representation across human and robot interactions while suppressing egocentric camera motion. To improve failure modeling, we further perform Group Relative Policy Optimization (GRPO) post-training on perturbed actions derived from successful demonstrations, guided by action alignment and physical plausibility rewards, enabling the model to learn plausible failure dynamics without paired ground-truth videos. Extensive experiments demonstrate that HF-WM achieves state-of-the-art performance in action-conditioned video generation and exhibits strong utility for policy evaluation in downstream robotic tasks.