Unassigned · 2026-10-01

Beyond Prediction: Steering VLM Agents with Retrospective World Modeling

Yongjiang Liu, Jie Zhang, Haoyue Zhang, Jingcai Guo, Deze Zeng, Song Guo

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose Retrospective World Modeling (RWM) for VLM agents: instead of only predicting future states from actions, the agent also infers a retrospective attribution P(â_t|s_t,s_{t+1}) for which action best explains an observed transition. They define a Self-Consistency Reward (SCR) that scores how consistent the executed action is with retrospective explanation, and integrate it into PPO for dense intrinsic feedback and improved robustness/generalization over prospective-only world modeling baselines.

Why it matters

Forward-only simulation can produce plausible but physically inconsistent behavior because it lacks a verification mechanism for whether an executed action causally matches the observed transition. By using retrospective attribution and SCR as dense transition-level feedback, the approach aims to steer policies toward actions that can be consistently explained by observed state changes, improving robustness and generalization across multiple agentic tasks.

Method

Introduce Retrospective World Modeling (RWM) that reasons backward over belief transitions (ŝ_{t-1}, ŝ_t) to infer retrospectively attributed actions â_{t-1}~πθ(·|ŝ_{t-1},ŝ_t).
Formulate Self-Consistency Reward (SCR) via discriminative retrospective action scoring over the action space and compute a normalized margin between executed action score and the strongest alternative.
Integrate SCR into reinforcement learning with PPO, using delayed assignment of r_i^t ← SCR(ŝ_t, ŝ_{t+1}, a_t) and dense reward integration with Bi-Level GAE (RWM-Full).

Limitation

The authors state two limitations: (1) in environments with extreme stochastic dynamics, the inverse transition mapping can become ambiguous, introducing noise into SCR; (2) scaling RWM to extremely long-horizon tasks has practical hardware challenges because RL training is memory-intensive and can encounter Out-Of-Memory (OOM) bottlenecks.

Abstract (from arXiv)

Equipping VLM agents with world modeling capabilities has shown strong potential for complex reasoning and long-horizon planning, while reducing the dependence of policy learning on costly real-world interactions. Existing methods mainly rely on prospective simulation to predict the consequences of candidate actions. However, this forward-only paradigm focuses on what will happen next and provides limited constraints for verifying whether an action is causally consistent with the observed state transition, which can lead to plausible-looking but physically incoherent behaviors. In this paper, we challenge the view of world modeling as only prospective prediction and introduce Retrospective World Modeling, a new agent learning paradigm that enables agents to reason backward by estimating the retrospective attribution distribution $P(\hat{a}{t}|s_t, s{t+1})$ for the action that most likely caused a given transition. Based on this capability, we formulate the Self-Consistency Reward (SCR), an intrinsic signal that measures the probabilistic consistency between the policy action and the retrospective explanation. Integrating SCR into reinforcement learning provides dense transition-level feedback and steers agents toward behaviors that are both task-effective and physically grounded. Extensive experiments across diverse agentic tasks show that our method substantially improves policy robustness and generalization over prospective-only world modeling baselines.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC

preprint