Latent Dynamics · 2026-10-28

Vision-Language Agents Can Learn in Their Own Latent Worlds

Haoxi Li, Sikai Bai, Jianfei Ma, Qinglin Hou, Xiaocheng Lu, Tao Han, Jie Zhang, Song Guo

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Dwell is a model-based RL framework that reframes a pretrained vision-language model (VLM) as a recurrent latent transition operator. It uses separate lightweight LoRA adapters for world modeling and policy while sharing the same latent state interface, enabling online learning from real interaction plus latent imagination. Across five interactive visual environments, the authors report the strongest overall performance among trainable methods while reducing imagination compute.

Why it matters

The paper targets efficient online policy improvement with pretrained VLMs, where world-model prediction must adapt as policy updates shift the interaction distribution. Dwell aims to address this by learning recurrent latent dynamics and reward prediction in a compact space, and by using the VLM’s own representations for both world modeling and policy, so imagined latent states can be consumed directly for planning.

Method

Recast a pretrained VLM as a recurrent latent transition operator with posterior inference (observation-conditioned) and prior prediction (action-conditioned) producing latent slots.
Use separate lightweight LoRA adapters for the world model and for the policy on a shared frozen VLM backbone, so predicted latent states can be fed directly to the policy during imagination.
Train with real interactions (posterior learning, dynamics and reward learning) and latent imagination (model-based latent planning with PPO), iterating between acting and updating.

Limitation

The authors report results under a “reported training protocol” and matched environmental feedback and interaction budget, but provide no stated guarantee beyond those settings; they also adapt LeWorldModel and PPO/GRPO under matched training and evaluation settings.

Abstract (from arXiv)

Learning from experience is essential for building the next generation of agents that continually expand their capabilities. Pretrained vision-language models (VLMs) already internalize rich world knowledge, yet this knowledge is rarely leveraged to drive policy improvement through online interaction. Realizing this potential efficiently remains challenging, as policy updates shift the agent's interaction distribution, requiring the world model to adapt accordingly, while continually updating large pretrained models is computationally expensive. To address this challenge, we introduce Dwell, a model-based reinforcement learning (RL) framework that reframes a VLM as a recurrent latent transition operator, endowing it with the ability to predict, adjust, and act in latent space. Leveraging the VLM's own representations and its dual role as both world model and policy, the agent continually learns from real interactions to anticipate and evaluate action outcomes, and from latent imagination to improve its behavior. Across five interactive visual environments, Dwell achieves the strongest overall performance among trainable methods while substantially reducing the compute required for imagination, showing that effective world models for VLM agents need not reconstruct the future; they need to predict futures that are useful for action. Our code is publicly available at https://github.com/Tom111-lhx/Dwell

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC

preprint