Driving · 2026-10-01
Uruqi: Learning Spatial Cognition from Visual Experience
Shichao Li, Meiqi Wang, Fei Su, Zhicheng Zhao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Uruqi trains vision-language models with dense multi-turn supervision on continuous visual experience. Each training episode is compiled from synthesized motif-driven camera trajectories into interleaved tasks: self-motion tracking (M1), persistent object mapping (M2), and operations over spatial state (M3). URUQI Syn-8B improves accuracy from 15.84% to 47.73% on the Uruqi benchmark; URUQI-SI-Mix-8B reaches 50.41% and transfers to external spatial benchmarks using only synthesized training data.
Why it matters
The authors target two atomic spatial failures in VLMs—tracking self-motion and mapping the world during motion—by making motion-to-spatial-state transitions explicit during training. They compile each synthesized continuous trajectory into temporally grounded, interleaved supervision so models learn connected updates across viewpoints, not isolated QA. They report gains on their Uruqi benchmark and transfer improvements on VSI-Bench, MMSI-Bench, and MindCube-tiny.
Method
- Generate 11,738 motif-driven camera trajectories over 3D scenes (rendered in OmniGibson) and compile each into temporally grounded multi-turn QA supervision.
- Interleave three linked supervision stages per episode: self-motion tracking (relative translation/rotation), persistent object mapping (cross-view localization with visibility/history), and operations over spatial state (reference-frame transforms, hypothetical/composed motion updates, metric/directional comparisons).
- Post-train VLMs with autoregressive next-token prediction over answer tokens, using the same evolving episode context that includes images and preceding QA exchanges.
Abstract (from arXiv)
Spatial intelligence requires maintaining a coherent understanding of the world as the embodied agent moves. Like humans, the agent must use its own motion to interpret changes across observations and update object locations and spatial relations accordingly. Despite spatial post-training having substantially broadened the spatial intelligence of vision-language models (VLMs), they still struggle with two atomic spatial capabilities: tracking self-motion and mapping the surrounding world during motion. To address this gap, we provide dense multi-turn supervision over interleaved atomic capabilities within each training episode, mimicking the visual experience of a continuously moving agent that reasons as it observes. To scale this up, we synthesize 11,738 motif-driven camera trajectories over a broad range of 3D scenes, supporting self-motion tracking, persistent object mapping, and rich spatial operations within each visual experience. By training models to reason over these atomic questions, our URUQI$_{\mathrm{Syn}}$-8B improves accuracy from 15.84% to 47.73% on our Uruqi benchmark comprising 52k questions across 2.7k episodes. URUQI-SI-Mix-8B further reaches 50.41%, comparable to the 50.08% achieved by GPT-6 Astra. Trained solely on our synthesized data, URUQI$_{\mathrm{Syn}}$-8B achieves an average relative accuracy improvement of 17.13% over its InternVL3-8B backbone across three external spatial benchmarks. These results highlight continuous visual experience as a scalable source of supervision for developing spatial cognition in VLMs.
Related papers
- Spatial Memory Intelligence: Endowing World Models with Understanding-Driven Long-Term Memory
- WOVEN: Weaving Visual World Modeling into Multimodal LLMs
- InsightMap: Structured Spatial Modeling for Embodied Multimodal Reasoning
- EmbodiedWorld-200K: Toward Embodied World Agents via Embodied-Planning Dataset and Interactive World Models
- G$^2$WAM: World Action Models for Efficient and Geometry Grounded Image-Goal Navigation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC
Uruqi: Learning Spatial Cognition from Visual Experience
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2609.39195
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Spatial intelligence requires maintaining a coherent understanding of the world as the embodied agent moves. Like humans, the agent must use its own motion to interpret changes across observations and update object locations and spatial relations accordingly. Despite spatial post-training having substantially broadened the spatial intelligence of vision-language models (VLMs), they still struggle with two atomic spatial capabilities: tracking self-motion and mapping the surrounding world during motion. To address this gap, we provide dense multi-turn supervision over interleaved atomic capabilities within each training episode, mimicking the visual experience of a continuously moving agent that reasons as it observes. To scale this up, we synthesize 11,738 motif-driven camera trajectories over a broad range of 3D scenes, supporting self-motion tracking, persistent object mapping, and rich spatial operations within each visual experience. By training models to reason over these atomic questions, our URUQI$_{\mathrm{Syn}}$-8B improves accuracy from 15.84% to 47.73% on our Uruqi benchmark comprising 52k questions across 2.7k episodes. URUQI-SI-Mix-8B further reaches 50.41%, comparable to the 50.08% achieved by GPT-6 Astra. Trained solely on our synthesized data, URUQI$_{\mathrm{Syn}}$-8B achieves an average relative accuracy improvement of 17.13% over its InternVL3-8B backbone across three external spatial benchmarks. These results highlight continuous visual experience as a scalable source of supervision for developing spatial cognition in VLMs.