Video World Models · 2026-10-23
DRILL: Training World Models to Improve Policies, Not Predict Pixels
Zhaolu Kang, Tailong Luo, Lei Wei, Guangyuan Dong, Sihan Liu, Yingjie He, Siheng Wang, Shuaibo Li, Rongchao Zhang, Zhen Tian, Richeng Xuan, Zhichao Hu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DRILL trains video world models for downstream control by updating the model to maximize the policy return after an inner GRPO update on imagined rollouts, instead of optimizing one-step pixel prediction. DRILL-VWI is a closed-form surrogate with no extra simulator rollouts; DRILL-IMG is a meta-gradient method differentiating through the inner update with truncated Hessian-vector products. Across manipulation simulators, DRILL-IMG improves mean success rate while increasing one-step pixel MSE.
Why it matters
The authors argue that in imagined-rollout RL, the world model’s role is to produce learning signals, not visually faithful next frames. They show that lower pixel MSE can yield worse policy gradients, and propose DRILL to directly align the world-model objective with downstream return.
Method
- Propose DRILL, a bilevel framework: inner loop performs one GRPO update in imagination using the world model; outer loop updates the world model so the post-update policy achieves higher return in the target simulator.
- DRILL-VWI replaces pixel MSE with a signed visitation-weighted surrogate reweighting errors by policy visitation, group-relative GRPO advantage, and model confidence, requiring no additional simulator rollouts over a WMPO-style pipeline.
- DRILL-IMG estimates the downstream-return meta-gradient by differentiating through the inner GRPO update using truncated BPTT and Hessian-vector products, grounding the outer return gradient with a small simulator rollout batch.
Abstract (from arXiv)
World models are increasingly used to reduce environment interaction when post-training vision-language-action (VLA) policies with reinforcement learning. Yet they are typically trained to predict pixels one step ahead, even though their role here is not merely to render visually faithful videos but to produce learning signals from imagined rollouts that improve the policy. We show that this mismatch is substantial: lower pixel MSE can produce worse policy gradients, and a world model trained for policy utility can yield better downstream policies despite 17% higher pixel MSE on the next frame. We propose the Downstream-Return Imagination Learning Loop (DRILL), a bilevel framework that updates the world model to maximize the policy's return after an inner GRPO update on imagined rollouts. DRILL has two instantiations. DRILL-VWI is a closed-form surrogate, requiring no additional simulator rollouts, that reweights prediction errors by policy visitation, the group-relative GRPO advantage, and model confidence. DRILL-IMG is a full meta-gradient method that differentiates through the inner update using truncated Hessian-vector products. We show that DRILL-VWI recovers the local one-step component of the DRILL-IMG meta-gradient under score-compatibility and small-residual assumptions. Across five manipulation simulators and two VLA backbones, DRILL-IMG raises mean success rate by 13.1 points over WMPO and 7.9 points over RLVR-World, and complements RLVR pretraining rather than competing with it. More broadly, we believe video world models for VLA reinforcement learning can be productively trained by the policy gradients they produce, not only by pixel-level prediction.
Related papers
- WoVR+: World Model as Reliable Simulator for Versatile VLA Policy Optimization with RL
- WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
- PLaW-VLA: Predictive Latent World Modeling for Vision-Language-Action Policies
- LaLa-World: Scaling Latent World Models with Natural Language Actions
- Forward Dynamics Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC
DRILL: Training World Models to Improve Policies, Not Predict Pixels
Preprint posted: 2026-10-22 · Original record ↗
- arxiv
- 2610.30814
Explicit source versions (1)
- v1 · 2026-10-22
- Preprint available
Source date: 2026-10-22 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World models are increasingly used to reduce environment interaction when post-training vision-language-action (VLA) policies with reinforcement learning. Yet they are typically trained to predict pixels one step ahead, even though their role here is not merely to render visually faithful videos but to produce learning signals from imagined rollouts that improve the policy. We show that this mismatch is substantial: lower pixel MSE can produce worse policy gradients, and a world model trained for policy utility can yield better downstream policies despite 17% higher pixel MSE on the next frame. We propose the Downstream-Return Imagination Learning Loop (DRILL), a bilevel framework that updates the world model to maximize the policy's return after an inner GRPO update on imagined rollouts. DRILL has two instantiations. DRILL-VWI is a closed-form surrogate, requiring no additional simulator rollouts, that reweights prediction errors by policy visitation, the group-relative GRPO advantage, and model confidence. DRILL-IMG is a full meta-gradient method that differentiates through the inner update using truncated Hessian-vector products. We show that DRILL-VWI recovers the local one-step component of the DRILL-IMG meta-gradient under score-compatibility and small-residual assumptions. Across five manipulation simulators and two VLA backbones, DRILL-IMG raises mean success rate by 13.1 points over WMPO and 7.9 points over RLVR-World, and complements RLVR pretraining rather than competing with it. More broadly, we believe video world models for VLA reinforcement learning can be productively trained by the policy gradients they produce, not only by pixel-level prediction.