Latent Dynamics · 2026-10-01

Asking the World: Generalist Physical Reasoning through Agentic World Modeling and Probing

Shenxiang Zeng, Chen Yang, Peiyao Chen, Guohui Zhang, Jiansheng Fan, Chen Wang

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Asking the World (ATW) is a training-free physical reasoning agent that builds an executable world from video (World Modeling) and then probes it with simulations and interventions to obtain evidence for questions (World Probing). It uses PolyWorld Engine, a Warp-based multiphysics simulator, and CEM system identification to calibrate task-relevant dynamics. Using Gemini-3-Flash, ATW improves accuracy on CLEVRER and ContPhy and on three real-world tasks.

Why it matters

The authors argue that direct VLM inference and fixed tool pipelines fail on complex physical tasks without explicit modeling and validation. ATW instead treats simulation as a persistent, question-directed workspace: the agent decides what world to construct, which physical hypotheses to instantiate, and how to probe via executed counterfactuals and interventions, grounded by CEM-fitted dynamics.

Method

Two-stage adaptive agent: World Modeling revises perceptual/physical assumptions and commits an executable world; World Probing queries, simulates, intervenes, and branches on the committed world without reopening World Modeling.
PolyWorld Engine: Warp-based multiphysics simulator supporting rigid bodies, soft bodies, cloth, ropes, fluids, and coupled interactions, with provenance-tracking agent-facing operations.
CEM-based system identification fits exposed parameters by sampling and minimizing trajectory discrepancy against visual evidence; calibration can revise representation or object binding before committing the shared world.

Abstract (from arXiv)

Physical reasoning from video requires inferring latent physical properties and dynamics beyond direct observation. Direct VLM inference remains unreliable on complex physical tasks without explicit modeling and validation, while predefined tool pipelines rely on task- and domain-specific priors that limit generalization across materials, dynamics, and reasoning tasks. We introduce Asking the World (ATW), a generalist agent that constructs and interrogates task-relevant executable worlds through two adaptive stages: World Modeling calibrates a world from video, while World Probing queries, simulates, and intervenes on it to obtain question-relevant evidence. Rather than prescribing the operations in either stage, ATW determines how to model and probe according to the scene and question. We develop PolyWorld Engine, a lightweight and highly programmable Warp-based multiphysics simulator for constructing and probing worlds with rigid bodies, soft bodies, cloth, ropes, fluids, and their coupled interactions. CEM-based system identification recovers task-relevant dynamics during World Modeling. The resulting world becomes an active workspace for question-directed physical experiments rather than a predetermined downstream tool. We evaluate ATW on CLEVRER, ContPhy, and three real-world scenarios. Using Gemini-3-Flash as its base VLM, ATW achieves 80.82% overall per-question accuracy on CLEVRER, improving direct Gemini-3-Flash by 46.50 points, GPT-5.5 by 13.58 points, and PhysMind by 8.27 points. On ContPhy, it reaches 70.56% overall accuracy, surpassing Gemini-3-Flash by 28.10 points and GPT-5.5 by 3.53 points. Across the three real-world scenarios, ATW achieves 71.67% accuracy, 28.33 points above GPT-5.5. These results establish agentic world modeling and probing as an effective, execution-grounded approach to generalist physical reasoning.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC

preprint