Latent Dynamics · 2026-10-20
AnyGoal: On World Model Generalization Beyond Expert Rollouts
Heejeong Nam, Divyat Mahajan, Quentin Le Lidec, Randall Balestriero, David Klindt
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AnyGoal benchmarks goal-shifted task variants to test whether world models generalize beyond expert rollouts. Tasks preserve marginal support of states, actions, and physical factors but change joint state–action combinations that are underrepresented in expert trajectories. The authors report broad failures across multiple environments and world-model families, including both generative world models and JEPA-style architectures.
Why it matters
World models are used for planning by simulating action-conditioned dynamics. The paper argues that in-distribution evaluation is insufficient when training data induce strong state–action correlations, so dynamics may fail on under-covered recombinations even if individual factors and actions are familiar. AnyGoal provides a controlled benchmark and evaluation to diagnose compositional/generalization failures in the combinatorial gap where test joint combinations are missing from the training joint support.
Method
- Define compositional generalization via a combinatorial space over physical factors and actions, and target the combinatorial gap where test joint combinations are absent from training joint support.
- Construct AnyGoal variants from base tasks (TwoRoom, Push-T, DMC Manipulator Insert-Peg, OGBench Cube) that pass factor-wise support checks while changing joint combinations; evaluate only as a test set.
- Evaluate base-trained models on both rollout prediction error (MSE over horizons) and downstream MPC planning success using standardized interfaces; also use oracle-state predictors to isolate dynamics vs representation failures.
Abstract (from arXiv)
The goal of a world model is to capture the dynamics of an environment. This has the great appeal that it allows agents to plan and simulate a course of actions with the help of a world model. For these uses to be reliable, however, the learned dynamics must behave well beyond the specific state-action combinations observed in expert rollouts. Despite this, the relationship between data coverage, dynamics generalization, and downstream planning performance remains underexplored. We study this relationship through the lens of compositional generalization: can a world model recombine familiar state and action factors into valid dynamics? To answer this, we introduce AnyGoal, a benchmark of goal-shifted task variants built from widely used world-model environments, together with an open-source library. Each task is designed to be within reach of a world model that has learned the underlying dynamics: individual state factors and actions remain within their training support, but their combinations are underrepresented in expert trajectories. Across multiple environments and world model families, we find that current models consistently exhibit failures under these recombinations. By examining dynamics and representation failures across model families, interaction regimes, and evaluation metrics, we study where current world models generalize and where challenges remain for goal-shifted prediction and planning. Together, this framework, benchmark, and analysis provide a basis for evaluating progress toward world models that generalize to new planning tasks.
Related papers
- An Introduction to World Models for Generalist Agents
- One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
- The Planning Limits of Latent World Models
- From Single-Actor Environments to Multi-Actor Event Worlds: Can LLMs Be Event World Models?
- How Much Planning Is Enough? Reducing Search and Computation in World-Model Planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-19 · Indexed by Pub-AI: 2026-10-20 UTC
AnyGoal: On World Model Generalization Beyond Expert Rollouts
Preprint posted: 2026-10-19 · Original record ↗
- arxiv
- 2610.26844
Explicit source versions (1)
- v1 · 2026-10-19
- Preprint available
Source date: 2026-10-19 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
The goal of a world model is to capture the dynamics of an environment. This has the great appeal that it allows agents to plan and simulate a course of actions with the help of a world model. For these uses to be reliable, however, the learned dynamics must behave well beyond the specific state-action combinations observed in expert rollouts. Despite this, the relationship between data coverage, dynamics generalization, and downstream planning performance remains underexplored. We study this relationship through the lens of compositional generalization: can a world model recombine familiar state and action factors into valid dynamics? To answer this, we introduce AnyGoal, a benchmark of goal-shifted task variants built from widely used world-model environments, together with an open-source library. Each task is designed to be within reach of a world model that has learned the underlying dynamics: individual state factors and actions remain within their training support, but their combinations are underrepresented in expert trajectories. Across multiple environments and world model families, we find that current models consistently exhibit failures under these recombinations. By examining dynamics and representation failures across model families, interaction regimes, and evaluation metrics, we study where current world models generalize and where challenges remain for goal-shifted prediction and planning. Together, this framework, benchmark, and analysis provide a basis for evaluating progress toward world models that generalize to new planning tasks.