Latent Dynamics · 2026-10-12
When Generative Models Act as Simulators
Sevan Brodjian, Michael Hobley, Pietro Perona
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors test when a generative model can act like a simulator via “editability”: after learning from observations only, can they map a simulator state edit into the model’s latent state so the next-step output matches the simulator’s edited rendering? They compare probe-based representation steering vs direct inverse mapping in Othello and Rayworld, finding environment-dependent success: steering degrades under rule simplification in Othello, while probes fail in Rayworld’s continuous setting yet inverse maps can still edit successfully.
Why it matters
This work isolates whether learned, decodable internal variables are also controllable in the simulator sense—whether editing model latents yields the same next-step outputs as editing a ground-truth simulator state. It provides an evaluation framework and shows that “decodable” does not imply “editable,” with different behavior across discrete board games and continuous partially observed physics-like environments.
Method
- Train a generative model to predict next observations from past observations only, then learn an external mapping from simulator state to latent state using paired simulator-state/latent data.
- Evaluate two editing mechanisms: representation steering (decoding simulator state via probes, then steering latent using probe directions/gradients) vs inverse mapping (fit a direct map from simulator state to latent and overwrite a residual point).
- Measure decodability with probe skill and editability with Edit Index (directional correctness on edited output support) and Edit Fidelity (overall prediction accuracy relative to unedited and edited targets).
Limitation
The paper measures editability only on the next prediction (one-step), and for transformer models persistent edits are hard; their main evaluation scores each edit on the next prediction rather than rollout persistence.
Abstract (from arXiv)
A simulator summarizes an environment in a state that users can read and change. Generative models learn representations of the world without explicit specification, but offer no built-in way to edit that world's state. We investigate the conditions under which a generative model can act as a simulator through $\textit{editability}$: whether changing a model's latent representation produces the next-step outputs we get by editing a simulator's state. Plainly stated, our goal is to find a mapping between a generative model's latent state and the simulator state, where the model is trained on observations alone and the mapping is learned afterward. We test two kinds of mappings, both learned from observations paired with simulator states. In one, we train probes to decode the simulator state from the model, then steer along them. In the other, we learn the mapping directly with an inverse map, which can succeed even where steering fails. We study both in two contrasting environments: the discrete board game Othello, and Rayworld, our own continuous world of moving discs seen through partial renderings, closer to physical scenes and video. We find that which mapping succeeds depends strongly on the environment, even though the state is decodable in every variant. In Othello, editability falls smoothly as the game is simplified, and even small rule changes have large consequences. In Rayworld, probes fail to steer along the simulator's continuous state, yet an inverse map sets the model from that state anyway. A model can therefore be driven by a simulator it may have never learned.
Related papers
- Learning Interactive Real-World Simulators
- Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
- WorldFoundry: Towards Unified Infrastructure for World Intelligence
- AgentGarten: Code Worlds for Evolving Agents
- Generating Critic-Guided Synthetic Experience in Latent World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-12 UTC
When Generative Models Act as Simulators
Preprint posted: 2026-10-09 · Original record ↗
- arxiv
- 2610.12847
Explicit source versions (1)
- v1 · 2026-10-09
- Preprint available
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
A simulator summarizes an environment in a state that users can read and change. Generative models learn representations of the world without explicit specification, but offer no built-in way to edit that world's state. We investigate the conditions under which a generative model can act as a simulator through $\textit{editability}$: whether changing a model's latent representation produces the next-step outputs we get by editing a simulator's state. Plainly stated, our goal is to find a mapping between a generative model's latent state and the simulator state, where the model is trained on observations alone and the mapping is learned afterward. We test two kinds of mappings, both learned from observations paired with simulator states. In one, we train probes to decode the simulator state from the model, then steer along them. In the other, we learn the mapping directly with an inverse map, which can succeed even where steering fails. We study both in two contrasting environments: the discrete board game Othello, and Rayworld, our own continuous world of moving discs seen through partial renderings, closer to physical scenes and video. We find that which mapping succeeds depends strongly on the environment, even though the state is decodable in every variant. In Othello, editability falls smoothly as the game is simplified, and even small rule changes have large consequences. In Rayworld, probes fail to steer along the simulator's continuous state, yet an inverse map sets the model from that state anyway. A model can therefore be driven by a simulator it may have never learned.
