Video World Models · 2026-10-02
Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana, Qinrong Cui, Erland Hilman Fuadi, Zayd M. K. Zuhri, Nanda Ryaas Absar, Ahmed Elshabrawy, Wilfried Ariel Mulyawan, Shoubin Yu, Yue Zhang, Mohit Bansal, Alham Fikri Aji
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Ego2Act evaluates whether video generation models can produce goal-directed egocentric manipulation videos with complete multi-step physical execution. The benchmark has 110 real-world tasks (2,640 videos) and introduces Ego2ActJudge, a reference-free, rubric-based evaluator that scores task completion and physics plausibility. The authors find models often skip dependent steps and fail fine-grained contact/persistent state modeling.
Why it matters
The authors argue that video models used as world simulators must generate rollouts that are functionally valid: intermediate actions must reach the goal with physically plausible state changes. Ego2Act targets this by using outcome goals with multi-step dependencies and a reference-free evaluation pipeline that aligns with human judgment, exposing gaps that shorter-horizon or fidelity-only benchmarks miss.
Method
- Ego2Act benchmark: 110 real-world egocentric tasks pairing an initial scene image with a high-level outcome goal; scored for task completion and physical plausibility over generated videos.
- Ego2ActJudge: reference-free VQA/rubric pipeline that derives observable subgoals from (scene, goal) and scores via ordered Task gates (T1–T3) and Physics gates (P1–P4) with early exit.
- Evaluate 6 video generation models on 480p inputs with per-video anonymous human scoring on a 25-case (600-video) panel, plus alignment comparisons against multiple automated baselines.
Limitation
Ego2ActJudge evaluates what a rollout visibly shows (subgoal completion via physically plausible interactions) and does not test whether a passing rollout helps downstream learning or planning.
Abstract (from arXiv)
Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilities is crucial, existing benchmarks focus mainly on single short actions or step-by-step instructions. This leaves multi-step physical reasoning underexplored, especially in egocentric video generation that requires planning to simulate proper execution to accomplish high-level goals by carrying out multiple real-world manipulations. We introduce Ego2Act, a goal-directed benchmark featuring 2,640 videos from 110 real-world tasks across day-to-day settings, varying object clutter and multi-step complexity. Given an initial scene image and a high-level goal, Ego2Act evaluates whether video generation models can produce realistic egocentric videos of a hand manipulating objects to carry out the task. To support scalable evaluation, we also introduce Ego2ActJudge, a reference-free evaluation pipeline that achieves better task completion and physics plausibility evaluation alignment with human consensus compared to relevant baselines. Our findings reveal that models' generated simulations often skip or partially execute steps, leaving later steps missing dependent states, which leads to unfulfilled goal. Furthermore, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and persistent world modeling. We hope Ego2Act provides a rigorous testbed for advancing video models toward physically plausible, goal-directed simulation.
Related papers
- Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
- EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
- World Action Learning via Interaction-Centric Spectral Latent Guidance
- Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
- CausalAlign: Causal Alignment of Video Generation Models for Consistent Ego-Motion
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC
Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
Preprint posted: 2026-10-01 · Original record ↗
- arxiv
- 2610.01092
Explicit source versions (1)
- v1 · 2026-10-01
- Preprint available
Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilities is crucial, existing benchmarks focus mainly on single short actions or step-by-step instructions. This leaves multi-step physical reasoning underexplored, especially in egocentric video generation that requires planning to simulate proper execution to accomplish high-level goals by carrying out multiple real-world manipulations. We introduce Ego2Act, a goal-directed benchmark featuring 2,640 videos from 110 real-world tasks across day-to-day settings, varying object clutter and multi-step complexity. Given an initial scene image and a high-level goal, Ego2Act evaluates whether video generation models can produce realistic egocentric videos of a hand manipulating objects to carry out the task. To support scalable evaluation, we also introduce Ego2ActJudge, a reference-free evaluation pipeline that achieves better task completion and physics plausibility evaluation alignment with human consensus compared to relevant baselines. Our findings reveal that models' generated simulations often skip or partially execute steps, leaving later steps missing dependent states, which leads to unfulfilled goal. Furthermore, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and persistent world modeling. We hope Ego2Act provides a rigorous testbed for advancing video models toward physically plausible, goal-directed simulation.