Driving · 2026-10-15
Model What Matters: Decision-Guided Selective World Modeling for Language Agents
Maolin Ran, Minda Chen, Di Huang, Wei Xiong, Fei Huang, Weiwen Liu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DGS-WM (Decision-Guided Selective World Modeling) is an RL + world-modeling co-training method for language agents. It selects decision-relevant parts of each environmental observation as world-modeling targets (Decision-Guided Observation Selection) and down-weights targets the model already predicts confidently (Confidence-Adaptive Weighting). The auxiliary world-modeling loss is jointly optimized with the RL loss, improving over full-observation world modeling (ECHO).
Why it matters
The authors argue that treating complete environmental observations uniformly as world-modeling targets wastes capacity and can add decision-irrelevant noise. DGS-WM refines supervision granularity within observations and adapts auxiliary loss strength to target mastery, aiming for more stable and effective RL+WM co-training for text agents.
Method
- Joint RL and auxiliary world modeling: optimize the same policy model with task RL loss plus a selective world-modeling loss (λWM scales auxiliary strength).
- Decision-Guided Observation Selection: split an observation into candidate units and use the sampling policy’s average token log-likelihood of the true next response to score support; retain units above a relative threshold τt=μt+κσt.
- Confidence-Adaptive Weighting: compute target prediction confidence qt=exp(−ℓt) from target negative log-likelihood, then weight supervision per target as wt=stopgrad((1−qt)^γ), so high-confidence targets get smaller weights.
Limitation
The paper text states no limitations.
Abstract (from arXiv)
Reinforcement learning provides a direct mechanism for optimizing task objectives in large language model-driven agents, but task rewards are typically sparse trajectory-level signals that only evaluate task success, without directly supervising action-conditioned environment transitions, leaving the model with limited explicit understanding of action consequences and environment dynamics. Recent work introduces auxiliary supervision through policy and world modeling co-training, yet typically treats complete environmental observations uniformly as world modeling targets, overlooking the varying decision relevance of observation content. We propose DGS-WM (Decision-Guided Selective World Modeling), a decision-guided policy-world modeling co-training approach for language agents. DGS-WM employs a Decision-Guided Observation Selection component that filters observation units with decision value as world modeling targets based on their support for subsequent policy responses, and a Confidence-Adaptive Weighting component that adaptively adjusts supervision strength according to the model's prediction confidence on selected targets. Together, these components form a decision-aware auxiliary world modeling loss, jointly optimized with the RL loss on the same policy model. Across multiple benchmarks covering single-hop/multi-hop question answering and interactive web tasks, DGS-WM achieves consistent improvements under different model scales and RL algorithms, demonstrating more stable and effective performance compared to full-observation world modeling methods.
Related papers
- DRILL: Training World Models to Improve Policies, Not Predict Pixels
- An Introduction to World Models for Generalist Agents
- AnyGoal: On World Model Generalization Beyond Expert Rollouts
- How To Train Your World Model: Fine-tuning vs RAG for LM-based World Modeling
- World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
Model What Matters: Decision-Guided Selective World Modeling for Language Agents
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.19642
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Reinforcement learning provides a direct mechanism for optimizing task objectives in large language model-driven agents, but task rewards are typically sparse trajectory-level signals that only evaluate task success, without directly supervising action-conditioned environment transitions, leaving the model with limited explicit understanding of action consequences and environment dynamics. Recent work introduces auxiliary supervision through policy and world modeling co-training, yet typically treats complete environmental observations uniformly as world modeling targets, overlooking the varying decision relevance of observation content. We propose DGS-WM (Decision-Guided Selective World Modeling), a decision-guided policy-world modeling co-training approach for language agents. DGS-WM employs a Decision-Guided Observation Selection component that filters observation units with decision value as world modeling targets based on their support for subsequent policy responses, and a Confidence-Adaptive Weighting component that adaptively adjusts supervision strength according to the model's prediction confidence on selected targets. Together, these components form a decision-aware auxiliary world modeling loss, jointly optimized with the RL loss on the same policy model. Across multiple benchmarks covering single-hop/multi-hop question answering and interactive web tasks, DGS-WM achieves consistent improvements under different model scales and RL algorithms, demonstrating more stable and effective performance compared to full-observation world modeling methods.