Video World Models · 2026-10-09

AgentGarten: Code Worlds for Evolving Agents

Jiawei Chi, Shangchen Miao, Zhiyuan Shi, Kailu Wu, Hanyang Wang, Weiliang Chen, Qiyu Dai, Jinshan Ren, Jun Gao, Mingsheng Long, Yueqi Duan, Jiangran Lyu, Jialong Wu, Fangfu Liu

arXiv:2610.12374 PDF Code

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AgentGarten is a framework for “code worlds” that combine simulator/game-engine programs with a shared real-time neural renderer. Engines maintain persistent state and execute explicit interaction rules, while the renderer generates visual observations from structured conditions (depth or normals) via a geometry-conditioned autoregressive video model distilled with Adversarial Forcing (distribution matching + exact replay + real-data adversarial supervision). Agents can perceive, act, and improve across rounds via distilled playbooks.

Why it matters

The authors target a bottleneck in scaling interactive environments that must be both faithful (consistent state, rules, dynamics) and realistic (visual observations matching real-world distributions). AgentGarten separates explicit, inspectable dynamics (engine/state) from real-time neural rendering (visual observations), enabling multiple worlds to share one renderer. Their distillation method (Adversarial Forcing) aims to keep visual quality and differentiability through history updates while supporting real-time block-causal generation for interactive agents.

Method

Build code worlds by coupling scene programs to simulators/game engines with persistent state and program-defined interaction rules; a shared neural renderer synthesizes agent observations from structured conditions via a unified interface.
Train a geometry-conditioned autoregressive video model as the renderer by adapting a pretrained bidirectional video model to block-causal generation using teacher-forcing adaptation.
Distill the renderer with Adversarial Forcing: distribution matching distillation on self-rollouts, exact replay with bitwise-identical recomputation for history gradients, and real-data adversarial training with exact R1/R2 regularization (avoiding double-backward through fused attention kernels).

Abstract (from arXiv)

Interactive virtual worlds allow agents to learn through exploration and interaction. What agents can learn is bounded by the environments they practice in, which must be faithful, with consistent state, rules, and dynamics, and realistic, with observations that follow the real-world visual distributions. Achieving both across diverse worlds remains a bottleneck. We introduce AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments. Its simulation backends maintain persistent world state and execute program-defined interaction rules, while the renderer generates visual observations from structured conditions exported through a common interface. To build the neural renderer, we adapt a pretrained video model to geometry conditions, distill it with our proposed Adversarial Forcing, and optimize inference for real-time interaction. Adversarial Forcing makes history prefilling differentiable through exact replay, so that losses on later predictions update how the renderer encodes prior observations, and adds real-data adversarial supervision to improve its visual quality. In AgentGarten, agents perceive the world through visual observations, interact with it in real time, and improve by distilling each round of experience into playbooks that subsequent agents inherit and refine. Our empirical study demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared with millions for a conventional reinforcement learning counterpart. As new worlds can be written as code and rendered through the same interface, environments can scale in both number and difficulty alongside their agents, a step toward agents that keep evolving through interactive experience.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-09 UTC

preprint