Latent Dynamics · 2026-10-22
Self-Improving Agents with World Model Predictive Paths
Haori Lu, Tianyu Gao, Yaoyao Liu, Yingying Li
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose a test-time, MPPI-inspired framework for LLM agents: at each step, the agent proposes multiple candidate actions, uses a world model to predict one-step outcomes, scores each outcome for task progress and risk, and applies MPPI-style (Gibbs) weights over predictive costs to guide a final controller to execute one action. A dual-branch experience memory improves both candidate generation/selection and next-observation prediction.
Why it matters
Language environments lack explicit dynamics and costs, so the method couples world-model prediction with decision-making. The MPPI-style weighting selects among imagined alternatives without executing them, and the dual-branch memory aims to improve both candidate coverage/selection and the accuracy of next-observation predictions over interaction, targeting a bounded predict–select–replan loop.
Method
- MPPI-inspired decision loop: generate up to K candidate actions, predict one-step next observations with a world model, evaluate task progress and risk to form costs, and compute MPPI-style Gibbs weights to guide the final action choice.
- Evaluator-based cost for language tasks: since language environments provide no explicit dynamics or costs, the evaluator assigns bounded predictive costs using progress, information value, and consistency with evidence.
- Dual-branch experience memory: a planner branch refines candidate generation and action selection using success/failure summaries, while a world model branch improves next-observation prediction using episodic/semantic memory updates after real interaction; model parameters are frozen.
Limitation
The paper does not make any explicit statement of limitations in the provided text beyond its regret decomposition sources (limited candidate coverage, errors in action selection, and inaccurate next-observation predictions).
Abstract (from arXiv)
Large language model (LLM) agents are increasingly expected to interact with environments and improve their decisions during deployment. However, how anticipating future outcomes and drawing on accumulated interaction experience can jointly support decision-making remains underexplored. We propose a test-time framework, inspired by Model Predictive Path Integral (MPPI) control, that combines multi-branch predictive analysis with experience-based adaptation. At each step, the agent proposes multiple candidate actions and imagines their outcomes with a world model, so that alternatives can be compared without being executed in the environment. Since language environments provide neither explicit dynamics nor costs, the predicted outcomes are evaluated for task progress and risk, and MPPI-style weights over the resulting costs guide the final decision. A dual-branch experience memory enables self-improvement: the planner branch refines candidate generation and action selection, while the world model branch improves next-observation prediction. Experiments on ALFWorld and ScienceWorld across three backbones demonstrate the strong overall performance of our method and the effectiveness of experience-guided predictive control. Our code will be made publicly available.
Related papers
- Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks
- EvoBelief: Falsifiable World Models for Self-Improving LLM Agents
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling
- JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shifts
- MemoWM: How World Models Change What Agents Need to Remember
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-22 UTC
Self-Improving Agents with World Model Predictive Paths
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.28945
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Large language model (LLM) agents are increasingly expected to interact with environments and improve their decisions during deployment. However, how anticipating future outcomes and drawing on accumulated interaction experience can jointly support decision-making remains underexplored. We propose a test-time framework, inspired by Model Predictive Path Integral (MPPI) control, that combines multi-branch predictive analysis with experience-based adaptation. At each step, the agent proposes multiple candidate actions and imagines their outcomes with a world model, so that alternatives can be compared without being executed in the environment. Since language environments provide neither explicit dynamics nor costs, the predicted outcomes are evaluated for task progress and risk, and MPPI-style weights over the resulting costs guide the final decision. A dual-branch experience memory enables self-improvement: the planner branch refines candidate generation and action selection, while the world model branch improves next-observation prediction. Experiments on ALFWorld and ScienceWorld across three backbones demonstrate the strong overall performance of our method and the effectiveness of experience-guided predictive control. Our code will be made publicly available.