Driving · 2026-11-11

Game-Enhanced Embodied Learning with Event and Foresight Pretraining

Lianlei Shan, Han Jin, Yaunchao Ma, Xinpeng Chen, Linhao Zhou, Xin Bin

arXiv:2611.13819PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors report event-and-foresight pretraining for InternVLA-A1.5 using 394 human gameplay recordings across 20 games. After 12,400 game-pretraining updates, they compare a transferred checkpoint against the public base using the same suite-specific robot fine-tuning. Across four LIBERO suites, aggregate success increases from 1,718/2,000 (85.90%) to 1,811/2,000 (90.55%) (+4.65 pp), with heterogeneous task outcomes.

Why it matters

It evaluates whether adding a full game-interaction pretraining stage (keyboard/mouse event prediction plus history-conditioned visual foresight) can improve downstream robot manipulation performance for InternVLA-A1.5, under a fixed downstream protocol. The paper also reports where gains and regressions occur across suites and tasks rather than assuming uniform improvement.

Method

  • Event-and-foresight pretraining for InternVLA-A1.5: next-event prediction from recorded keyboard/mouse events plus history-conditioned visual supervision via frozen video modules; attention masks block teacher-forced future actions from entering foresight queries.
  • Coverage-preserving game sampling: coverage recovery restores 3,038 legal anchors for 6,681 otherwise uncovered eligible action ordinals, raising coverage from 99.94% to 100%, while constructing one-second action and video targets.
  • Checkpoint transfer and evaluation: transferred pretraining checkpoint (after 12,400 updates) is fine-tuned under the same suite-specific LIBERO robot protocol and compared to a public base under identical downstream settings.

Limitation

The comparison includes additional optimization and does not isolate the effects of game data, the two objectives, or training variability; the results also concern the selected human corpus, model, and simulator protocol and do not establish consistent improvement across all tasks, games, architectures, or real robots.

Abstract (from arXiv)

Human gameplay supplies paired observations, actions, and visual consequences that can supplement robot-policy training. We study event-and-foresight pretraining for InternVLA-A1.5, combining recorded keyboard/mouse event prediction with history-conditioned visual supervision through a frozen video model. Coverage-preserving anchor selection constructs one-second targets, and an attention mask prevents teacher-forced future actions from entering foresight queries. The selected corpus contains 394 recordings across 20 games, with 223.36 hours of recorded timestamp span and 2,072,084 training anchors. After 12,400 game-pretraining updates, we compare the resulting initialization with the public base under the same suite-specific robot fine-tuning protocol. Across four LIBERO suites, aggregate success increases from 1,718/2,000 (85.90%) to 1,811/2,000 (90.55%), a gain of 4.65 percentage points. Suite gains are 9.2, 6.8, 0.2, and 2.4 points on Spatial, Object, Goal, and Long. The 40-task breakdown shows 24 improvements, 10 declines, and six ties. These results support the practical value of the complete game-pretraining stage in this evaluated setting, while revealing heterogeneous task outcomes. The comparison includes additional optimization and does not isolate the effects of game data, the two objectives, or training variability.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-11 · Indexed by Pub-AI: 2026-11-11 UTC

preprint

Game-Enhanced Embodied Learning with Event and Foresight Pretraining

Lianlei Shan, Han Jin, Yaunchao Ma, Xinpeng Chen, Linhao Zhou, Xin Bin

First announced by arXiv: 2026-11-11 · Original record ↗

arxiv
2611.13819
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-11 · First detected: 2026-11-11 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.13819v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Human gameplay supplies paired observations, actions, and visual consequences that can supplement robot-policy training. We study event-and-foresight pretraining for InternVLA-A1.5, combining recorded keyboard/mouse event prediction with history-conditioned visual supervision through a frozen video model. Coverage-preserving anchor selection constructs one-second targets, and an attention mask prevents teacher-forced future actions from entering foresight queries. The selected corpus contains 394 recordings across 20 games, with 223.36 hours of recorded timestamp span and 2,072,084 training anchors. After 12,400 game-pretraining updates, we compare the resulting initialization with the public base under the same suite-specific robot fine-tuning protocol. Across four LIBERO suites, aggregate success increases from 1,718/2,000 (85.90%) to 1,811/2,000 (90.55%), a gain of 4.65 percentage points. Suite gains are 9.2, 6.8, 0.2, and 2.4 points on Spatial, Object, Goal, and Long. The 40-task breakdown shows 24 improvements, 10 declines, and six ties. These results support the practical value of the complete game-pretraining stage in this evaluated setting, while revealing heterogeneous task outcomes. The comparison includes additional optimization and does not isolate the effects of game data, the two objectives, or training variability.