Latent Dynamics · 2026-11-03
Roz: A Taste Test for World Model Latents
Kenneth Ge, Dhruv Sheth, Kunvar Thaman
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Roz evaluates whether latent dynamics world models encode object-relevant physical properties beyond image similarity. The authors introduce Roz, a training-free, model-agnostic benchmark using two evaluation methods (contrastive and continuous nearest-neighbor) with 10 contrastive tasks, 3 continuous tasks, and 1734 scenarios. They show that popular models do well on endpoint/image robustness but fail on tasks where history-revealed properties (e.g., friction, mass, deformation) matter.
Why it matters
The paper argues that existing evaluations (linear probes and downstream task performance) are opaque, resource-intensive, and do not directly test whether latent representations capture dynamics-relevant properties. Roz provides a probe-free, training-free, model-agnostic interface that only requires ingesting still frames or video, extracting latents, and applying a distance function—supporting faster and more comparable evaluation across latent world models.
Method
- Defines what dynamics-based latents should encode (object state plus revealed properties like mass, friction, restitution, deformation) with properties such as path independence, memory, and temporal consistency.
- Introduces two distance-based evaluation strategies: contrastive triplets (odd-one-out with strict inequalities) and continuous nearest-neighbor sweeps (checking that neighboring physical changes map to neighboring latent distances).
- Creates a model-agnostic interface: requires latents from any latent world model (direct or via hidden-state extraction) and a practitioner-supplied distance function (tests both L1 and cosine similarity).
Limitation
Roz requires a well-formed distance function for each model, and while the authors validate L1 and cosine similarity with controls and agreement, they note that fully validating a distance function is nontrivial and it is not yet established what these metrics ought to be. The readout choice (layer/token pooling) is also per-model and affects continuous-task scores.
Abstract (from arXiv)
State of the art latent dynamics world models achieve impressive performance for robot planning. But do they actually encode the physics of the world? In this paper, we show that popular world models today achieve strong results on tasks that reduce to image similarity but forget key details about objects in the scene like mass, deformation, friction, and bounciness. We achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance roughly improves with training steps, while not necessarily being correlated with reconstruction MSE. We argue current model failures are a symptom of evaluation strategies (probes and planners) that are opaque, model-specific, and high-compute. Roz requires only a model that can ingest still frames or video, a way to extract latents, and a distance function between them, and it can be run in one pass. We hope future evaluations will adopt Roz's evaluation techniques. More broadly, we hope Roz can be a case study for a future where benchmarks allow rapid iteration and comparison across models.
Related papers
- T2-z: Test-Time Optimization of Environment Latents for Robotic World Models
- The Planning Limits of Latent World Models
- RoboJEPA: Scaling Robotic Latent World Models
- LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
- What Limits Cross-Task Generalization in World Action Models?
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC
Roz: A Taste Test for World Model Latents
First announced by arXiv: 2026-11-03 · Original record ↗
- arxiv
- 2611.00612
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.00612v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
State of the art latent dynamics world models achieve impressive performance for robot planning. But do they actually encode the physics of the world? In this paper, we show that popular world models today achieve strong results on tasks that reduce to image similarity but forget key details about objects in the scene like mass, deformation, friction, and bounciness. We achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance roughly improves with training steps, while not necessarily being correlated with reconstruction MSE. We argue current model failures are a symptom of evaluation strategies (probes and planners) that are opaque, model-specific, and high-compute. Roz requires only a model that can ingest still frames or video, a way to extract latents, and a distance function between them, and it can be run in one pass. We hope future evaluations will adopt Roz's evaluation techniques. More broadly, we hope Roz can be a case study for a future where benchmarks allow rapid iteration and comparison across models.
