Robotics · 2026-10-16
T2-z: Test-Time Optimization of Environment Latents for Robotic World Models
Chenyuan Zhou, Yining Hong, Huang Huang, Yunfan Jiang, Wenlong Huang, David Held, Vitor Campagnolo Guizilini, Yue Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, Yejin Choi
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
T2-z adds an environment latent z_e to an action-conditioned robotic world model and trains it with alternating optimization: learn general action dynamics with z_e fixed, then optimize z_e with the world model frozen. At test time, the world model stays frozen while a new z_e is optimized from observed interactions to adapt to unseen environments. The authors report improved object/physical prediction error and action success in simulation and real-world tasks.
Why it matters
Robots must handle hidden environmental factors that are not directly observable or explicitly specified. The authors propose optimizing an environment latent at test time—without predefined physical factors or a learned history encoder—so the world model becomes an environment-adapted simulator via test-time optimization.
Method
- Augment an action-conditioned robotic world model with a randomly initialized environment latent z_e that conditions future visual predictions; z_e has no supervision about which factors to represent or their values.
- Training alternates two stages under a flow-matching objective: (1) world-model optimization with z_e fixed, and (2) environment-latent optimization with the world model frozen.
- At test time, freeze the world model, initialize a new z_test latent, and optimize it from observed test-time interaction trajectories; reuse the optimized latent to predict outcomes for candidate actions.
Abstract (from arXiv)
Predicting how the world responds to an action requires more than what can be seen: robots must account for environmental factors that are neither directly observable nor explicitly specified, from physical properties such as friction and mass to human-defined environmental rules. We introduce T2-Z for test-time optimization of environment latents, which augments a robotic world model with a learnable environment latent to condition world predictions. This compact latent captures underlying environmental variation without supervision of what environmental factors exist or what values they take. During training, T2-Z alternates between two stages: a world-model optimization stage that fixes the latents and learns general action-conditioned dynamics across environments; and an environment-latent optimization stage that freezes the world model and optimizes the conditioning latents, allowing latent structure to emerge purely through world prediction. At test time, the world model remains frozen and a newly initialized environment latent is optimized from one or more test-time interactions, adapting the model to previously unseen environmental configurations. Across 7 simulated settings and 4 real-world robotic tasks, T2-Z reduces object prediction error metric from 15.20 to 7.00 and raises action success from 33.73% to 72.96%. Its emergent latents capture structured representations of individual and compounded factors across diverse visual scenes. Ablations further show that T2-Z can efficiently infer environment latents from as little as a single test-time interaction trajectory.
Related papers
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-15 · Indexed by Pub-AI: 2026-10-16 UTC
T2-z: Test-Time Optimization of Environment Latents for Robotic World Models
Preprint posted: 2026-10-15 · Original record ↗
- arxiv
- 2610.22364
Explicit source versions (1)
- v1 · 2026-10-15
- Preprint available
Source date: 2026-10-15 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Predicting how the world responds to an action requires more than what can be seen: robots must account for environmental factors that are neither directly observable nor explicitly specified, from physical properties such as friction and mass to human-defined environmental rules. We introduce T2-Z for test-time optimization of environment latents, which augments a robotic world model with a learnable environment latent to condition world predictions. This compact latent captures underlying environmental variation without supervision of what environmental factors exist or what values they take. During training, T2-Z alternates between two stages: a world-model optimization stage that fixes the latents and learns general action-conditioned dynamics across environments; and an environment-latent optimization stage that freezes the world model and optimizes the conditioning latents, allowing latent structure to emerge purely through world prediction. At test time, the world model remains frozen and a newly initialized environment latent is optimized from one or more test-time interactions, adapting the model to previously unseen environmental configurations. Across 7 simulated settings and 4 real-world robotic tasks, T2-Z reduces object prediction error metric from 15.20 to 7.00 and raises action success from 33.73% to 72.96%. Its emergent latents capture structured representations of individual and compounded factors across diverse visual scenes. Ablations further show that T2-Z can efficiently infer environment latents from as little as a single test-time interaction trajectory.