Video World Models · 2026-11-09
Video-Coder: Reinforcing Agentic Coding for Video Generation
Chaoyang Wang, Sicheng Gao, Junan Hu, Kunyang Li, Dongyang Chen, Yiyang Zhou, Huaxiu Yao, Yong Jae Lee, Radu Timofte, Yuzhang Shang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Video-Coder is a code-driven video generation agent. It writes and executes Blender Python code to build a dynamic 3D scene and renders a “Draft Video” that provides structure and motion signals for a fixed downstream video generator. The method trains with SFT on Video-Coder-SFT-6k and then harness-native agentic RL on Video-Coder-RL-2k using execution- and video-based rewards with an annealing schedule.
Why it matters
The authors target controllability and physical plausibility in video generation by making structure and motion explicit and verifiable through executable code. They introduce a dedicated interactive harness for code execution and feedback, plus datasets and a dual-reward RL scheme to improve multi-turn code revision and final video faithfulness on physics-focused benchmarks.
Method
- Train a multimodal coding agent that iteratively writes Blender Python code, runs it in a Video-Coder Harness, inspects execution/visual feedback, and revises until the Draft Video matches the prompt.
- Curate two datasets: Video-Coder-SFT-6k (6k samples for supervised fine-tuning) and Video-Coder-RL-2k (2k prompt–reference video pairs for agentic RL), created via multi-turn trajectories and ground-truth video synthesis with MiniMax-H3.
- Optimize with harness-native agentic RL using dual rewards: execution-based reward (tool call correctness and successful execution) and video-based reward (evaluated consistency with prompt/reference), with an annealing strategy to gradually reduce the execution reward over training.
Abstract (from arXiv)
Recent years have witnessed remarkable progress in video generation in terms of visual quality and temporal consistency. However, generating videos with precise spatial structure, coherent motion dynamics, and controllable scene composition remains challenging. In this paper, we introduce Video-Coder, the first attempt to train a code-driven video generation agent, which autonomously writes and executes code to construct a dynamic 3D scene (Draft Video) that provides structure and motion signals to downstream video generators. To support this paradigm, we design a dedicated data pipeline and curate two high-quality datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, which contain diverse prompts and ground-truth reference videos. We further develop the Video- Coder Harness, a unified interactive environment integrating code execution, 3D scene rendering, and execution feedback for agent training and inference. We then train Video-Coder with SFT fol- lowed by harness-native agentic reinforcement learning with dual reward feedback, which combines execution-based and video-based rewards. Additionally, we design an annealing training strategy that gradually reduces the execution reward over time, allowing the model to rely more on the video reward in the later training stages. Experiments show that Video-Coder achieves significant gains, improving Minimax-H3 by approximately 7 points on PhyGenBench and 24 points on VBench-2.0. We hope this work can serve as an open foundation for coding agents in video generation, and we fully open-source our data, models, and code
Related papers
- Video Generation Meets Agent Harness: Agentic Video Generation, Agentic World Modeling and Recursive Self-Improvement
- BWM-Forge: Agentic World Programming for Embodied Intelligence
- Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
- WorldDraft: If Code Builds the World, Can Video Models Just Render It?
- Rethinking Video Autoencoders for Interactive World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-06 · Indexed by Pub-AI: 2026-11-09 UTC
Video-Coder: Reinforcing Agentic Coding for Video Generation
Preprint posted: 2026-11-06 · Original record ↗
- arxiv
- 2611.09894
Explicit source versions (1)
- v1 · 2026-11-06
- Preprint available
Source date: 2026-11-06 · First detected: 2026-11-09 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.09894v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent years have witnessed remarkable progress in video generation in terms of visual quality and temporal consistency. However, generating videos with precise spatial structure, coherent motion dynamics, and controllable scene composition remains challenging. In this paper, we introduce Video-Coder, the first attempt to train a code-driven video generation agent, which autonomously writes and executes code to construct a dynamic 3D scene (Draft Video) that provides structure and motion signals to downstream video generators. To support this paradigm, we design a dedicated data pipeline and curate two high-quality datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, which contain diverse prompts and ground-truth reference videos. We further develop the Video- Coder Harness, a unified interactive environment integrating code execution, 3D scene rendering, and execution feedback for agent training and inference. We then train Video-Coder with SFT fol- lowed by harness-native agentic reinforcement learning with dual reward feedback, which combines execution-based and video-based rewards. Additionally, we design an annealing training strategy that gradually reduces the execution reward over time, allowing the model to rely more on the video reward in the later training stages. Experiments show that Video-Coder achieves significant gains, improving Minimax-H3 by approximately 7 points on PhyGenBench and 24 points on VBench-2.0. We hope this work can serve as an open foundation for coding agents in video generation, and we fully open-source our data, models, and code
