Driving · 2026-10-30
TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents
Guangzhi Wang, Junfu Pu, Yiran Wang, Xingyilang Yin, Honghao Chen, Cheng Cheng, Lingen Li, Yinan Zhou, Jiarui Wu, Yinuo Jiang, Hongbo Jin, Xiaodong Cun, Ying Shan
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TaskFrame is a hierarchical planner–controller for Minecraft game agents. A vision-language task planner decomposes a high-level goal into an ordered sequence of atomic instructions once per task, using supervised learning and dependency-aware GRPO. Special goal-token hidden states are cached as goal embeddings and reused by a per-frame causal Transformer controller that outputs native keyboard/mouse actions and continuous camera motion every game step. Trained in three stages, it attains higher combat success than evaluated baselines.
Why it matters
The authors address a core issue in pixel-based game agents: task-level planning and frame-level control run at different frequencies and have different computational needs. By connecting cached goal embeddings from a once-per-task decomposition planner to a lightweight frame controller that acts at every step, the approach aims to preserve low-latency control while improving semantic task execution. The paper reports competitive instruction decomposition and higher closed-loop combat success on atomic Minecraft tasks.
Method
- Vision-language planner decomposes a high-level instruction into an ordered sequence of atomic goals, each terminated by a special goal token; the token hidden states are cached as continuous goal embeddings.
- Dependency-aware GRPO refinement: the authors use separate rewards for step-count agreement and dependency-aware ordering based on must-precede annotations, with Hungarian step alignment.
- Per-frame causal Transformer controller: conditions on the active goal embedding plus recent observations/action history to predict discrete keyboard/mouse-button actions and continuous camera yaw/pitch each game step; trained in three stages (planner, controller pretrain, joint alignment).
Limitation
The controller is trained on atomic segments and windows never cross an atomic-segment boundary; the authors note that an atomic one-second context can limit behaviors requiring longer-horizon exploration (e.g., the original stone task).
Abstract (from arXiv)
Building general game agents requires both high-level task reasoning and low-latency action control in rapidly changing visual environments. These capabilities favor models with substantially different computational profiles, making them difficult to realize within a single model. In this work, we present TaskFrame, a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard-and-mouse control. A vision-language task planner decomposes a high-level goal into an ordered sequence of atomic instructions and is optimized through supervised learning and dependency-aware GRPO. The hidden states of special goal tokens that terminate each atomic instruction provide continuous goal embeddings that are cached and reused across control steps. A causal Transformer controller combines the embedding of the active goal with recent observations to predict discrete keyboard and mouse-button actions and continuous camera motion at every game step. We train the system in three stages: planner training, controller pretraining, and joint alignment. Experiments in Minecraft show that TaskFrame is competitive with general-purpose vision-language models on instruction decomposition, and that in closed-loop evaluation the agent achieves a higher combat success rate than the evaluated baselines.
Related papers
- From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models
- AceAction-5M: Hierarchical Goals for Generalist Action Learning Across 3D Game Worlds
- Player-1: Learning to Play from Synthetic Experience
- Code to Control: Synthesizing Parameterized Reactive Controllers
- LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC
TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents
Preprint posted: 2026-10-29 · Original record ↗
- arxiv
- 2610.40276
Explicit source versions (1)
- v1 · 2026-10-29
- Preprint available
Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40276v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Building general game agents requires both high-level task reasoning and low-latency action control in rapidly changing visual environments. These capabilities favor models with substantially different computational profiles, making them difficult to realize within a single model. In this work, we present TaskFrame, a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard-and-mouse control. A vision-language task planner decomposes a high-level goal into an ordered sequence of atomic instructions and is optimized through supervised learning and dependency-aware GRPO. The hidden states of special goal tokens that terminate each atomic instruction provide continuous goal embeddings that are cached and reused across control steps. A causal Transformer controller combines the embedding of the active goal with recent observations to predict discrete keyboard and mouse-button actions and continuous camera motion at every game step. We train the system in three stages: planner training, controller pretraining, and joint alignment. Experiments in Minecraft show that TaskFrame is competitive with general-purpose vision-language models on instruction decomposition, and that in closed-loop evaluation the agent achieves a higher combat success rate than the evaluated baselines.
