Driving · 2026-10-12
Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing
Zheng Ren, Yanning Wang, Ziheng Huang, Fei Huang, Shiqi Wang, Ziqian Ye, Yang Yue
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
GeoSpatialBench evaluates spatial relation understanding and constrained spatial editing on real urban scenes. Using 28 formal relations across GeoJSON and rendered map images (and Together), the authors find a “spatial gap”: GeoJSON generally outperforms map images, Together does not consistently improve, and editing—especially for ordering—often fails due to constraint violations.
Why it matters
The benchmark tests whether models can not only answer spatial QA but also modify geometry while satisfying explicit constraints, with GIS-verified checking. The reported mismatch between recognition (QA) and intervention (editing) exposes limitations relevant to grounded world models and embodied spatial AI at geospatial scales.
Method
- GeoSpatialBench combines spatial QA and constrained geometric editing over 28 GIS-defined relations (topology, direction, metric, ordering) on city-scale scenes from OpenStreetMap.
- Each scene is available as GeoJSON, map image, or Together; QA is scored by accuracy and editing by Constraint Satisfaction Rate (CSR) (and raster-tolerant success for map-image editing).
- The authors evaluate nine models across six cities and analyze performance by relation family, relation, and representation; editing targets geometry/attributes while leaving other features unchanged.
Limitation
The authors report that their results do not establish generalization to unseen cities.
Abstract (from arXiv)
Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models. However, existing evaluations of multimodal large language models (MLLMs) for spatial reasoning rely predominantly on static question-answering (QA) over small-scale or synthetic scenes, rarely probing whether models can modify spatial configurations while satisfying explicit geometric constraints. We introduce GeoSpatialBench, a benchmark designed to evaluate an escalating progression of spatial cognition, ranging from spatial relation understanding and geometric reasoning to constrained spatial editing. Grounded in real-world urban data across six geographically diverse cities, GeoSpatialBench evaluates a comprehensive taxonomy of 28 topological, directional, metric, and ordering relations across structured GeoJSON and map image modalities, with ground-truth references and constraints verified via GIS operations. Our empirical evaluation across frontier MLLMs uncovers a stark spatial gap: structured GeoJSON consistently outperforms map images, with multimodal combinations offering no reliable gain over GeoJSON alone, and while models show reasonable competence on qualitative topological and ordering, performance drops on quantitative metric relations. Crucially, editing performance on ordering relations declines relative to QA, and most failed edits violate at least one specified spatial constraint. These results highlight a gap between passively recognizing spatial relations and actively manipulating geometric configurations. GeoSpatialBench provides an open, rigorous diagnostic testbed to benchmark these spatial limitations, laying essential empirical groundwork for physically grounded world models and embodied spatial AI.
Related papers
- GeoThinker-$F$: Endogenous Predictive World Representations for Fast Spatial Intelligence
- InsightMap: Structured Spatial Modeling for Embodied Multimodal Reasoning
- Uruqi: Learning Spatial Cognition from Visual Experience
- WOVEN: Weaving Visual World Modeling into Multimodal LLMs
- Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-12 UTC
Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing
Preprint posted: 2026-10-09 · Original record ↗
- arxiv
- 2610.12881
Explicit source versions (1)
- v1 · 2026-10-09
- Preprint available
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models. However, existing evaluations of multimodal large language models (MLLMs) for spatial reasoning rely predominantly on static question-answering (QA) over small-scale or synthetic scenes, rarely probing whether models can modify spatial configurations while satisfying explicit geometric constraints. We introduce GeoSpatialBench, a benchmark designed to evaluate an escalating progression of spatial cognition, ranging from spatial relation understanding and geometric reasoning to constrained spatial editing. Grounded in real-world urban data across six geographically diverse cities, GeoSpatialBench evaluates a comprehensive taxonomy of 28 topological, directional, metric, and ordering relations across structured GeoJSON and map image modalities, with ground-truth references and constraints verified via GIS operations. Our empirical evaluation across frontier MLLMs uncovers a stark spatial gap: structured GeoJSON consistently outperforms map images, with multimodal combinations offering no reliable gain over GeoJSON alone, and while models show reasonable competence on qualitative topological and ordering, performance drops on quantitative metric relations. Crucially, editing performance on ordering relations declines relative to QA, and most failed edits violate at least one specified spatial constraint. These results highlight a gap between passively recognizing spatial relations and actively manipulating geometric configurations. GeoSpatialBench provides an open, rigorous diagnostic testbed to benchmark these spatial limitations, laying essential empirical groundwork for physically grounded world models and embodied spatial AI.