Unassigned · 2026-11-10

From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models

Changran Liu, Qi Wu, Wenchao Ma, Sharon X. Huang, Haomiao Jiang

arXiv:2611.11136PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors present Script2Play, a framework that automatically generates game environments and expert demonstrations from skill specifications, using an LLM-powered coding agent plus programmatic teachers that act through keyboard and mouse. They post-train a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human demonstrations or reinforcement learning, and evaluate on multiple game benchmarks.

Why it matters

Collecting aligned human gameplay demonstrations for vision-language-action (VLA) instruction following is costly, and RL requires extensive interactive rollouts and reward design. The paper frames both skill-targeted game creation and expert demonstration generation as audited coding-agent tasks, turning privileged simulator state into verified, aligned instruction-frame-action supervision for post-training VLA policies, with reported cross-engine generalization.

Method

  • Use an LLM-powered coding agent to generate executable game environments and tasks targeting specified skills; games pass five automated audit gates before being used.
  • Generate demonstrations with programmatic experts using privileged simulator state (poses, target coordinates, enemy/projectile states, collision geometry) while acting only via standard keyboard and mouse.
  • Train Script2Play by post-training a pretrained vision-language model with behavior cloning on verified synthetic trajectories recorded as 200 ms action windows (h=0 in benchmarks).

Limitation

The paper reports that approximate hour equivalents for some baselines are “inferred from reported frame or token counts, not matched training budgets,” so reported training-scale comparisons may not reflect controlled data-efficiency.

Abstract (from arXiv)

Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models

Changran Liu, Qi Wu, Wenchao Ma, Sharon X. Huang, Haomiao Jiang

First announced by arXiv: 2026-11-10 · Original record ↗

arxiv
2611.11136
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11136v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.