Latent Dynamics · 2026-10-28

Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning

Zengqi Zhao, Zun Wang, Shoubin Yu, Mohit Bansal, Yue Zhang

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors argue that spatial reasoning over imagined states should not default to RGB. They introduce ImagiRep, a simulator-grounded benchmark with oracle RGB and six alternative representations of the same unobserved state, and show that accurate imagined RGB does not guarantee accurate spatial reasoning and that different tasks benefit from different representations. They further propose GEARS, a test-time adaptive representation selection strategy that outperforms existing methods.

Why it matters

This work isolates how representation choice affects spatial reasoning for unseen states. By providing controlled oracle vs estimated representations and evaluating multiple spatial question types, the authors show that “better-looking” imagined RGB can mislead downstream reasoning. The GEARS strategy targets this gap by gating/routing evidence and selecting representations suited to the question.

Method

ImagiRep: construct a simulator-grounded benchmark using Matterport3D geometry/poses, rendering oracle RGB plus six alternative representations (Depth, Surface Normals, Panorama, Top-down Geometry, Spatial Text, RGB Video) for the same target unobserved state.
Evaluate spatial reasoning under fixed observed views, fixed question, and varying representation inputs across two settings: Viewpoint Imagination and Cross-view Integration.
GEARS: at test time, gate by comparing reasoner answers with/without estimated endpoint RGB; if they disagree, route to select among evidence/representations before producing the final multiple-choice answer.

Limitation

The authors note that even with ground-truth endpoint views, substantial errors remain, suggesting that accurate imagined RGB alone is insufficient for reliable spatial reasoning.

Abstract (from arXiv)

Visual world models increasingly support spatial reasoning by generating RGB views of unobserved states. However, RGB has become the default interface largely because it is well suited for visual generation, rather than necessarily being the most effective representation for downstream reasoning. Different spatial questions may benefit from different representations of the same imagined state: top-down geometry can facilitate global distance comparisons, while panoramic views provide broader context for reasoning about open space. To isolate the role of representation in spatial imagination, we introduce ImagiRep, a simulator-grounded benchmark of 922 questions spanning seven question types across two settings, Viewpoint Imagination and Cross-view Integration, providing oracle RGB and six alternative representations (e.g., depth, panoramic views, and top-down geometry) of the same unobserved state. This controlled comparison reveals that even accurate imagined RGB does not guarantee accurate spatial reasoning, and different questions benefit from different representations. Similar trends hold for representations derived from world-model-estimated states. Notably, estimated RGB alone can mislead spatial reasoning, while alternative representations retain complementary benefits. Building on these findings, we introduce GEARS, a lightweight test-time strategy for adaptive representation selection, outperforming existing spatial imagination methods and showing that reasoning about unseen states requires more than generating RGB views.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC

preprint