Unassigned · 2026-11-10
From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models
Changran Liu, Qi Wu, Wenchao Ma, Sharon X. Huang, Haomiao Jiang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors present Script2Play, a framework that automatically generates game environments and expert demonstrations from skill specifications, using an LLM-powered coding agent plus programmatic teachers that act through keyboard and mouse. They post-train a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human demonstrations or reinforcement learning, and evaluate on multiple game benchmarks.
Why it matters
Collecting aligned human gameplay demonstrations for vision-language-action (VLA) instruction following is costly, and RL requires extensive interactive rollouts and reward design. The paper frames both skill-targeted game creation and expert demonstration generation as audited coding-agent tasks, turning privileged simulator state into verified, aligned instruction-frame-action supervision for post-training VLA policies, with reported cross-engine generalization.
Method
- Use an LLM-powered coding agent to generate executable game environments and tasks targeting specified skills; games pass five automated audit gates before being used.
- Generate demonstrations with programmatic experts using privileged simulator state (poses, target coordinates, enemy/projectile states, collision geometry) while acting only via standard keyboard and mouse.
- Train Script2Play by post-training a pretrained vision-language model with behavior cloning on verified synthetic trajectories recorded as 200 ms action windows (h=0 in benchmarks).
Limitation
The paper reports that approximate hour equivalents for some baselines are “inferred from reported frame or token counts, not matched training budgets,” so reported training-scale comparisons may not reflect controlled data-efficiency.
Abstract (from arXiv)
Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.
Related papers
- Player-1: Learning to Play from Synthetic Experience
- AceAction-5M: Hierarchical Goals for Generalist Action Learning Across 3D Game Worlds
- AgentGarten: Code Worlds for Evolving Agents
- TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents
- Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.11136
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11136v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.
