Latent Dynamics · 2026-11-04

FlowBench: Counterfactual Imagination Training for Spatial Intelligence

Yifan Liu, Baihan Zhang, Khai Loong Aw, Klemen Kotar, Daniel L. K. Yamins

arXiv:2611.03937PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

FlowBench defines Counterfactual Spatial Imagination (CSI): answering spatial questions about a scene after a hypothetical object motion without observing the outcome. It provides ~428K training examples (sim + real) and a fully automatic, simulator-derived pipeline with exact post-motion supervision (frames/flow). The authors report VLM “imagination gaps” and show Qwen3-VL-8B fine-tuning on FlowBench-Sim improves accuracy on both sim and real; they also show world-model training improves text-to-flow prediction.

Why it matters

The authors target a missing evaluation/training setup for spatial intelligence in VLMs and world models: counterfactual changes to spatial configuration under object motion, not just observed scenes or camera motion. FlowBench offers scalable counterfactual data generation (with exact simulator supervision for sim) and benchmark splits for measuring sim generalization and sim-to-real transfer, enabling targeted training for both perception-reasoning models and flow/frame world models.

Method

  • Formulate CSI for multi-choice visual QA: given (initial image I0, hypothetical motion tm, spatial question tq), predict the answer ta without observing the post-motion outcome.
  • Build FlowBench with a scalable automatic simulator pipeline: compute ground-truth answers from simulator metadata and attach exact post-motion supervision (outcome image, optical flow, depth, segmentation) plus reasoning annotations.
  • Evaluate both VLMs and world models: VLMs answer from (I0, tm, tq); world models are assessed via predicted optical flow and (optionally) generated outcome frames, including text-conditioned generation.

Limitation

Perception and imagination: 1A accuracy combines reading the scene with imagining the motion, and 2A separates the two only where the static task is easy; on FlowBench-Sim, static metric questions are themselves hard, so 1A there also measures metric perception. The authors also note that FlowBench-Real answers rely on estimated depth, masks, and tracks; absolute distances remain approximate.

Abstract (from arXiv)

Humans imagine scene changes under hypothetical motions, an ability central to spatial reasoning and robotic planning. However, this capability remains underexplored in vision-language models and world models. We formalize this capability as Counterfactual Spatial Imagination (CSI) and introduce FlowBench, a resource for training and evaluating both, with 428K training examples spanning simulated and real-world scenes, household environments and tabletop manipulation, and diverse question types. A scalable, fully automatic pipeline pairs each hypothetical motion with spatial questions and, in simulation, exact outcome frames and optical flow. For VLMs, proprietary models exhibit an imagination gap, performing worse when motion outcomes must be inferred than when observed, and open-source models perform near chance; fine-tuning Qwen3-VL-8B on FlowBench-Sim-Train raises its accuracy by 29.5 percentage points in simulation and 16.9 on real images, a strong sim-to-real transfer. For world models, training on FlowBench-Sim improves text-to-flow prediction for all four model families we test, raising the accuracy of Psi-0.6 from 52.1% to 61.7%. FlowBench thus serves as a training and evaluation resource for both. Code and datasets are released at https://github.com/liuyifan22/FlowBench.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-04 · Indexed by Pub-AI: 2026-11-04 UTC

preprint

FlowBench: Counterfactual Imagination Training for Spatial Intelligence

Yifan Liu, Baihan Zhang, Khai Loong Aw, Klemen Kotar, Daniel L. K. Yamins

First announced by arXiv: 2026-11-04 · Original record ↗

arxiv
2611.03937
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-04 · First detected: 2026-11-04 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.03937v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Humans imagine scene changes under hypothetical motions, an ability central to spatial reasoning and robotic planning. However, this capability remains underexplored in vision-language models and world models. We formalize this capability as Counterfactual Spatial Imagination (CSI) and introduce FlowBench, a resource for training and evaluating both, with 428K training examples spanning simulated and real-world scenes, household environments and tabletop manipulation, and diverse question types. A scalable, fully automatic pipeline pairs each hypothetical motion with spatial questions and, in simulation, exact outcome frames and optical flow. For VLMs, proprietary models exhibit an imagination gap, performing worse when motion outcomes must be inferred than when observed, and open-source models perform near chance; fine-tuning Qwen3-VL-8B on FlowBench-Sim-Train raises its accuracy by 29.5 percentage points in simulation and 16.9 on real images, a strong sim-to-real transfer. For world models, training on FlowBench-Sim improves text-to-flow prediction for all four model families we test, raising the accuracy of Psi-0.6 from 52.1% to 61.7%. FlowBench thus serves as a training and evaluation resource for both. Code and datasets are released at https://github.com/liuyifan22/FlowBench.