Driving · 2026-11-10
Learning to Act with World Models by Valuing What Comes Next
Chenghao Gao, Zhi Zhang, Dian Jin, Yihe Pan, Doudou Zhou
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
WM-PPO combines PPO’s value-based success estimation with a world model’s predicted next states. For each candidate action (executed or not), the world model predicts the next observation and the PPO critic values the predicted successor. Comparisons yield auxiliary policy advantages; the mean transition value is a critic target. A shared gate calibrates both channels using real branch outcomes.
Why it matters
Sparse terminal rewards make it hard to assign credit to intermediate steps in text environments. The authors’ key idea is to treat predicted dynamics as a way to evaluate how actions change the probability of eventual success, leveraging PPO’s critic value estimates. WM-PPO learns from agent interactions without teacher or privileged information and uses only the policy at inference, aiming to improve success with comparable real interaction budgets.
Method
- At each training state, the policy samples several candidate actions; it executes the first and uses the world model to predict next observations for other candidates too.
- Transition values are produced by using the critic on either predicted successors (or predicted endings), enabling candidate comparisons.
- Two learning channels: auxiliary advantages from comparisons (policy channel) and an auxiliary one-step critic target from the mean transition value (value channel), weighted by a shared gate calibrated to real branch outcomes.
Abstract (from arXiv)
A sparse terminal reward tells an agent little about its steps. Much of the missing information is implicit in the environment's dynamics, because each state transition can change the conditions for success long before the reward arrives. A world model trained on real transitions can learn to predict what each action changes, but a predicted observation does not by itself show whether success has become more likely. With a binary terminal reward, PPO's critic makes this judgment by estimating the current policy's probability of success, but only for states the agent reaches. Our method, WM-PPO, elegantly brings the two together. For each action the policy samples, executed or not, the world model predicts the next observation and the critic values the predicted successor. In the policy channel, comparisons among these values give the policy auxiliary advantages; in the value channel, their mean gives the critic an auxiliary target. A shared gate weights both channels according to how reliably comparisons with similar margins predicted real outcomes earlier in training. The policy, critic and world model all learn from the agent's own interaction, with no teacher or privileged information, and only the policy is used at inference. With a comparable number of real environment steps, WM-PPO raises success from PPO's 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld. It matches the strongest PPO refinements we tested on ALFWorld and exceeds the best of them by 6.5 points on ScienceWorld. The trained policy more often completes prerequisite steps before placing an object or committing to an answer.
Related papers
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling
- Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials
- How Much Can Best-of-N Help a World Action Model? Measuring Selectable Headroom and Realizing It in Closed Loop
- World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits
- Predict the Reward, Not the World: World Reward Models for ML Agents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
Learning to Act with World Models by Valuing What Comes Next
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.12200
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.12200v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
A sparse terminal reward tells an agent little about its steps. Much of the missing information is implicit in the environment's dynamics, because each state transition can change the conditions for success long before the reward arrives. A world model trained on real transitions can learn to predict what each action changes, but a predicted observation does not by itself show whether success has become more likely. With a binary terminal reward, PPO's critic makes this judgment by estimating the current policy's probability of success, but only for states the agent reaches. Our method, WM-PPO, elegantly brings the two together. For each action the policy samples, executed or not, the world model predicts the next observation and the critic values the predicted successor. In the policy channel, comparisons among these values give the policy auxiliary advantages; in the value channel, their mean gives the critic an auxiliary target. A shared gate weights both channels according to how reliably comparisons with similar margins predicted real outcomes earlier in training. The policy, critic and world model all learn from the agent's own interaction, with no teacher or privileged information, and only the policy is used at inference. With a comparable number of real environment steps, WM-PPO raises success from PPO's 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld. It matches the strongest PPO refinements we tested on ALFWorld and exceeds the best of them by 6.5 points on ScienceWorld. The trained policy more often completes prerequisite steps before placing an object or committing to an answer.
