Driving · 2026-10-15
GeoThinker-$F$: Endogenous Predictive World Representations for Fast Spatial Intelligence
Haoyuan Li, Qihang Cao, Huanyuan Zhou, Xin Hu, Sijia Chen, Zihan Guo, Kun Xiang, Yunshuang Nie, Tao Tang, Xinlong Zheng, Jia-Wang Bian, Xianming Liu, Xiaodan Liang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
GeoThinker-F removes runtime dependence on an external 3D geometry encoder for geometry-aware spatial intelligence. It trains lightweight Latent Frame Prediction (LFP) heads to predict latent current-geometry and next-frame semantic representations from intermediate causal MLLM hidden states, then integrates them via Spatial-Grounded Temporal (SGT) Fusion. The authors report up to a 3.6× inference speedup (1.45 s to 0.40 s on Qwen3-VL-8B) with competitive spatial reasoning performance.
Why it matters
By internalizing spatiotemporal perception into the MLLM latent space, GeoThinker-F targets the inference-time latency and compute overhead caused by running external 3D encoders. The method is trained with a two-stage pipeline that uses an offline geometry expert only as a teacher in Stage 1, then removes it for Stage 2 and inference, aiming for an improved accuracy–efficiency trade-off in spatial intelligence benchmarks and driving/robotics-related settings.
Method
- Freeze an MLLM backbone in Stage 1 and train lightweight Latent Frame Prediction (LFP) heads on intermediate layers using an offline 3D geometry expert (VGGT) and next-frame semantic targets.
- Remove the external expert in Stage 2 and jointly optimize the model so self-predicted geometry/future-semantic priors are routed into the MLLM through Spatial-Grounded Temporal (SGT) Fusion modules.
- Use dual-branch (geometry-prior and temporal-prior) integration with Layer-Interleaved Feature Injection to alternate/calibrate integration intensity across intermediate transformer layers.
Limitation
The authors observe limitations for endogenous feature prediction: (1) transient non-rigid dynamics (e.g., a translucent curtain fluttering) where GeoThinker-F fails to register moving geometric boundaries captured by VGGT; and (2) representation smoothness, where self-predicted features are visually coarser with higher spatial jitter due to token-level spatial discretization in Transformer architectures.
Abstract (from arXiv)
Existing geometry-aware Multimodal Large Language Models (MLLMs) increasingly rely on external 3D encoders to provide structural priors, but executing these additional encoder paths at inference introduces substantial latency and computational overhead. We propose GeoThinker-$F$, an efficient framework that internalizes spatiotemporal perception by transforming causal MLLM hidden states into predictive latent world representations. Our key insight is that intermediate semantic representations contain sufficient information to predict useful representations of current geometry and future semantics, without requiring an exact reproduction of features from a dedicated 3D encoder. To realize this idea, GeoThinker-$F$ adopts a decoupled two-stage training strategy. In the first stage, the MLLM backbone is frozen while lightweight Latent Frame Prediction (LFP) heads learn from an offline 3D geometry expert and next-frame semantic targets. In the second stage, the external expert is removed, and the model is jointly optimized to generate and selectively integrate its self-predicted priors through a specialized Spatial-Grounded Temporal (SGT) Fusion module. Extensive evaluations show that GeoThinker-$F$ eliminates runtime dependence on external 3D geometry encoders while maintaining competitive performance across diverse spatial intelligence benchmarks. It achieves up to a $3.6\times$ inference speedup, reducing latency from $1.45\,\mathrm{s}$ to $0.40\,\mathrm{s}$ while retaining strong spatial reasoning performance.
Related papers
- Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing
- Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models
- GeoState: Learning View-Decoupled Geometry States for 3D World Modeling
- Uruqi: Learning Spatial Cognition from Visual Experience
- Spatial Memory Intelligence: Endowing World Models with Understanding-Driven Long-Term Memory
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
GeoThinker-$F$: Endogenous Predictive World Representations for Fast Spatial Intelligence
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.20592
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Existing geometry-aware Multimodal Large Language Models (MLLMs) increasingly rely on external 3D encoders to provide structural priors, but executing these additional encoder paths at inference introduces substantial latency and computational overhead. We propose GeoThinker-$F$, an efficient framework that internalizes spatiotemporal perception by transforming causal MLLM hidden states into predictive latent world representations. Our key insight is that intermediate semantic representations contain sufficient information to predict useful representations of current geometry and future semantics, without requiring an exact reproduction of features from a dedicated 3D encoder. To realize this idea, GeoThinker-$F$ adopts a decoupled two-stage training strategy. In the first stage, the MLLM backbone is frozen while lightweight Latent Frame Prediction (LFP) heads learn from an offline 3D geometry expert and next-frame semantic targets. In the second stage, the external expert is removed, and the model is jointly optimized to generate and selectively integrate its self-predicted priors through a specialized Spatial-Grounded Temporal (SGT) Fusion module. Extensive evaluations show that GeoThinker-$F$ eliminates runtime dependence on external 3D geometry encoders while maintaining competitive performance across diverse spatial intelligence benchmarks. It achieves up to a $3.6\times$ inference speedup, reducing latency from $1.45\,\mathrm{s}$ to $0.40\,\mathrm{s}$ while retaining strong spatial reasoning performance.