Latent Dynamics · 2026-10-21

World Models with Configurable Perception

Yijun Shen, Delong Chen, Xianming Hu, Kai Zhang, Pascale Fung

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

ConfigWM is an action-conditioned world model for planning under dynamic distraction. It uses a language task query to configure perception and map visual embeddings into a task-specific predictive state space, suppressing task-irrelevant distractor motion. The learned dynamics are then rolled out in this space for MPC planning.

Why it matters

The authors target a limitation of task-agnostic world model encoders: they can be distracted by task-irrelevant information, making prediction and planning unreliable when multiple target-like objects compete. By conditioning state extraction on the task query, ConfigWM aims to learn representations and dynamics that preserve task-relevant content under increasing distraction.

Method

ConfigWM uses a configurable encoder conditioned on a language task query to map noisy visual observations into task-specific predictive states.
A frozen configurable encoder encodes current/next observations under the same query; an action-conditioned causal Transformer (with a trainable projector) learns dynamics in this task-relevant latent space.
Planning uses CEM within an MPC loop, scoring action sequences by terminal latent distance to the goal state encoded under the same query.

Limitation

The paper text does not state explicit limitations.

Abstract (from arXiv)

Real-world observations are noisy and often contain excessive task-irrelevant information. Effective world modeling and planning need to abstract observations into world state representations while filtering out irrelevant information. However, encoders in existing world models are typically non-adaptive, leading to state representations that are easily distracted by task-irrelevant information. As a result, the model may not selectively filter observations to cater to different task demands. To address this problem, we introduce ConfigWM, which realizes configurable perception through task-conditioned state extraction. Given a language description of the task at hand, ConfigWM uses it as a configuration signal to steer how visual embeddings are mapped into a task-specific state space, selectively preserving information relevant to the current task while suppressing task-irrelevant variations. Action-conditioned dynamics are then learned in this task-specific state space for planning. We evaluate ConfigWM in environments with increasing levels of distraction. While existing methods exhibit substantial declines in planning success rate as the environments become increasingly noisy, ConfigWM remains comparatively stable and outperforms the strongest competing method by 22.3% on average across diverse environments. Consistent gains in physical-quantity probing (+0.365 Pearson r) and distracted-to-clean retrieval (+75.5%) further indicate that ConfigWM learns states that preserve task-relevant information while remaining robust to distractors, supporting reliable world modeling and planning in noisy worlds. Project Page: https://configwm.github.io

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC

preprint