Latent Dynamics · 2026-11-04
FlowBench: Counterfactual Imagination Training for Spatial Intelligence
Yifan Liu, Baihan Zhang, Khai Loong Aw, Klemen Kotar, Daniel L. K. Yamins
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
FlowBench defines Counterfactual Spatial Imagination (CSI): answering spatial questions about a scene after a hypothetical object motion without observing the outcome. It provides ~428K training examples (sim + real) and a fully automatic, simulator-derived pipeline with exact post-motion supervision (frames/flow). The authors report VLM “imagination gaps” and show Qwen3-VL-8B fine-tuning on FlowBench-Sim improves accuracy on both sim and real; they also show world-model training improves text-to-flow prediction.
Why it matters
The authors target a missing evaluation/training setup for spatial intelligence in VLMs and world models: counterfactual changes to spatial configuration under object motion, not just observed scenes or camera motion. FlowBench offers scalable counterfactual data generation (with exact simulator supervision for sim) and benchmark splits for measuring sim generalization and sim-to-real transfer, enabling targeted training for both perception-reasoning models and flow/frame world models.
Method
- Formulate CSI for multi-choice visual QA: given (initial image I0, hypothetical motion tm, spatial question tq), predict the answer ta without observing the post-motion outcome.
- Build FlowBench with a scalable automatic simulator pipeline: compute ground-truth answers from simulator metadata and attach exact post-motion supervision (outcome image, optical flow, depth, segmentation) plus reasoning annotations.
- Evaluate both VLMs and world models: VLMs answer from (I0, tm, tq); world models are assessed via predicted optical flow and (optionally) generated outcome frames, including text-conditioned generation.
Limitation
Perception and imagination: 1A accuracy combines reading the scene with imagining the motion, and 2A separates the two only where the static task is easy; on FlowBench-Sim, static metric questions are themselves hard, so 1A there also measures metric perception. The authors also note that FlowBench-Real answers rely on estimated depth, masks, and tracks; absolute distances remain approximate.
Abstract (from arXiv)
Humans imagine scene changes under hypothetical motions, an ability central to spatial reasoning and robotic planning. However, this capability remains underexplored in vision-language models and world models. We formalize this capability as Counterfactual Spatial Imagination (CSI) and introduce FlowBench, a resource for training and evaluating both, with 428K training examples spanning simulated and real-world scenes, household environments and tabletop manipulation, and diverse question types. A scalable, fully automatic pipeline pairs each hypothetical motion with spatial questions and, in simulation, exact outcome frames and optical flow. For VLMs, proprietary models exhibit an imagination gap, performing worse when motion outcomes must be inferred than when observed, and open-source models perform near chance; fine-tuning Qwen3-VL-8B on FlowBench-Sim-Train raises its accuracy by 29.5 percentage points in simulation and 16.9 on real images, a strong sim-to-real transfer. For world models, training on FlowBench-Sim improves text-to-flow prediction for all four model families we test, raising the accuracy of Psi-0.6 from 52.1% to 61.7%. FlowBench thus serves as a training and evaluation resource for both. Code and datasets are released at https://github.com/liuyifan22/FlowBench.
Related papers
- Uruqi: Learning Spatial Cognition from Visual Experience
- GlanceWAM: Sparse Test-Time Imagination for World-Action Models
- CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness
- WorldMind: Test-Time Scaling for Visual Spatial Reasoning through Latent World Simulation
- $\Psi$-0.6: Adaptive Streams of Thought in a Unified Vision-Language World Model
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-04 · Indexed by Pub-AI: 2026-11-04 UTC
FlowBench: Counterfactual Imagination Training for Spatial Intelligence
First announced by arXiv: 2026-11-04 · Original record ↗
- arxiv
- 2611.03937
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-04 · First detected: 2026-11-04 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.03937v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Humans imagine scene changes under hypothetical motions, an ability central to spatial reasoning and robotic planning. However, this capability remains underexplored in vision-language models and world models. We formalize this capability as Counterfactual Spatial Imagination (CSI) and introduce FlowBench, a resource for training and evaluating both, with 428K training examples spanning simulated and real-world scenes, household environments and tabletop manipulation, and diverse question types. A scalable, fully automatic pipeline pairs each hypothetical motion with spatial questions and, in simulation, exact outcome frames and optical flow. For VLMs, proprietary models exhibit an imagination gap, performing worse when motion outcomes must be inferred than when observed, and open-source models perform near chance; fine-tuning Qwen3-VL-8B on FlowBench-Sim-Train raises its accuracy by 29.5 percentage points in simulation and 16.9 on real images, a strong sim-to-real transfer. For world models, training on FlowBench-Sim improves text-to-flow prediction for all four model families we test, raising the accuracy of Psi-0.6 from 52.1% to 61.7%. FlowBench thus serves as a training and evaluation resource for both. Code and datasets are released at https://github.com/liuyifan22/FlowBench.
