Video World Models · 2026-11-04
AceAction-5M: Hierarchical Goals for Generalist Action Learning Across 3D Game Worlds
Kailing Li, Ruoxuan Zhang, Zheyuan Zhou, Boyang Chu, Yingshan Lei, Hongxia Xie, Liang Du, Zixun Sun
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AceAction-5M is a multi-game dataset of human gameplay in 3D games, with hierarchical language annotations for complete tasks, temporally bounded stage-level subgoals, and local skill instructions aligned to frame-level keyboard-and-mouse actions. The authors build annotations by identifying subgoals from objective/interaction-target changes, grounding navigation goals to visually recognizable waypoints, and composing adjacent subgoals into task instructions.
Why it matters
The dataset provides structured supervision for multi-stage instruction following in 3D game environments, explicitly connecting overall task descriptions to ordered subgoals and fine-grained skill instructions with explicit stage boundaries and temporally aligned control inputs.
Method
- They annotate skill-based subgoals by grouping four consecutive actions with an observation, then using a vision-language model to predict skill category, start/end indices, and subgoal descriptions; segmentation changes when objective or interaction target changes.
- They construct navigation subgoals from arrival events at visually recognizable waypoints, assigning descriptive waypoint names and creating clips that end at the waypoint frame with sampled keyframes and full action sequences.
- They apply two-stage quality control: model-based verification using Qwen3.8-27B and GLM-5.3-FLASH across action/subgoal/overall-instruction levels, then human inspection of 500 trajectories (each reviewed by at least two reviewers).
Abstract (from arXiv)
Video games provide a practical setting for collecting vision-language-action (VLA) data, whose acquisition in physical environments is costly. Existing datasets, however, remain limited by restricted game coverage, sparse language supervision, or temporal hierarchies that do not necessarily reflect changes in task objectives. We introduce AceAction, a dataset of human gameplay across multiple 3D games for learning skill composition and subgoal transitions. AceAction organizes demonstrations around complete tasks, linking overall instructions, ordered subgoals, and local skill instructions to temporally aligned visual observations and keyboard-and-mouse actions. To construct these annotations, we develop a vision-language model pipeline that identifies subgoals from changes in objectives or interaction targets, anchors navigation goals to visually recognizable waypoints, and composes adjacent subgoals into task instructions. The resulting demonstrations cover navigation, reconnaissance, combat, interaction, and recovery, with explicit stage boundaries that connect task progression to fine-grained control. We assess annotation quality through model-based verification and human inspection. An audit of 500 trajectories yields a 95.8% pass rate across independent reviewer assessments. AceAction provides a data foundation for studying how agents execute, combine, and switch skills to follow complex instructions in dynamic game environments.
Related papers
- From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models
- Player-1: Learning to Play from Synthetic Experience
- PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence
- Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?
- TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-03 · Indexed by Pub-AI: 2026-11-04 UTC
AceAction-5M: Hierarchical Goals for Generalist Action Learning Across 3D Game Worlds
Preprint posted: 2026-11-03 · Original record ↗
- arxiv
- 2611.04241
Explicit source versions (1)
- v1 · 2026-11-03
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-04 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.04241v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video games provide a practical setting for collecting vision-language-action (VLA) data, whose acquisition in physical environments is costly. Existing datasets, however, remain limited by restricted game coverage, sparse language supervision, or temporal hierarchies that do not necessarily reflect changes in task objectives. We introduce AceAction, a dataset of human gameplay across multiple 3D games for learning skill composition and subgoal transitions. AceAction organizes demonstrations around complete tasks, linking overall instructions, ordered subgoals, and local skill instructions to temporally aligned visual observations and keyboard-and-mouse actions. To construct these annotations, we develop a vision-language model pipeline that identifies subgoals from changes in objectives or interaction targets, anchors navigation goals to visually recognizable waypoints, and composes adjacent subgoals into task instructions. The resulting demonstrations cover navigation, reconnaissance, combat, interaction, and recovery, with explicit stage boundaries that connect task progression to fine-grained control. We assess annotation quality through model-based verification and human inspection. An audit of 500 trajectories yields a 95.8% pass rate across independent reviewer assessments. AceAction provides a data foundation for studying how agents execute, combine, and switch skills to follow complex instructions in dynamic game environments.
