Driving · 2026-10-08
STRIKE: Learning Visual State Transitions for Physical World Modeling
Wenbin Teng, Tianshuo Xu, Depu Meng, Yuelei Li, Quentin Herau, Yihan Hu, Yajie Zhao, Wei Zhan
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
STRIKE learns physical world modeling via explicit visual state transitions instead of directly generating dense video. It builds event-aligned supervision by extracting observed visual states from training videos and pairing adjacent states with transition descriptions and elapsed times. A transition image model predicts the next state recursively under VLM-planned timed events; a separately trained video dynamics model then generates dense rollouts conditioned on those predicted timed states.
Why it matters
The authors argue that physical world modeling must predict interaction outcomes as relations between objects (visual states), not just produce coherent motion. STRIKE uses learned, time-conditioned visual state transitions as an intermediate representation and conditions a video generator on predicted states/timestamps, improving physical consistency and manipulation-video fidelity versus video-backbone baselines.
Method
- Build event-aligned transition supervision by extracting visual state frames from training videos and pairing adjacent selected states with transition descriptions and elapsed time offsets.
- Train a time-conditioned image transition model to predict the next visual state from (current image, local transition spec, elapsed time), then recursively compose predicted states during inference.
- Train a separate conditional video dynamics model to generate dense rollouts from the initial context conditioned on the predicted states and their temporal locations (implemented with CogVideoX and Wan backbones).
Abstract (from arXiv)
Physical world modeling requires predicting how interactions change a scene, not merely generating coherent motion. We propose STRIKE, a framework that separates visual state transition learning from dense video generation. We construct event-aligned supervision by extracting observed states from training videos and pairing them with transition descriptions and temporal offsets. An image-based transition model learns to predict the next scene configuration from the current image, a local transition specification, and elapsed time. At inference, a pretrained vision-language planner predicts time transition specifications, and recursive application of the learned transition model produces a sequence of future visual states. A separately trained dynamic model then generates the complete rollout conditioned on these states and their temporal locations. Experiments on Physics-IQ Verified, PhyGenBench, Pisa-Experiments, and RoboTwin2.0 show improvements of STRIKE over the corresponding video-backbone baselines in benchmark measures of physical consistency and manipulation-video fidelity. These results support learned visual state transitions as an effective intermediate representation for physical world modeling.
Related papers
- PhysiCalWorld: In-Context Physics Calibration for World Models
- Time-JEPA: Grounding Video World Models with Physical Time Series
- WOVEN: Weaving Visual World Modeling into Multimodal LLMs
- Momentum: Persistent Dynamics for Video World Models
- Beyond One-Step Accuracy: State-Affine Latent Transition for Reliable Visual Planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC
STRIKE: Learning Visual State Transitions for Physical World Modeling
Preprint posted: 2026-10-07 · Original record ↗
- arxiv
- 2610.09514
Explicit source versions (1)
- v1 · 2026-10-07
- Preprint available
Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Physical world modeling requires predicting how interactions change a scene, not merely generating coherent motion. We propose STRIKE, a framework that separates visual state transition learning from dense video generation. We construct event-aligned supervision by extracting observed states from training videos and pairing them with transition descriptions and temporal offsets. An image-based transition model learns to predict the next scene configuration from the current image, a local transition specification, and elapsed time. At inference, a pretrained vision-language planner predicts time transition specifications, and recursive application of the learned transition model produces a sequence of future visual states. A separately trained dynamic model then generates the complete rollout conditioned on these states and their temporal locations. Experiments on Physics-IQ Verified, PhyGenBench, Pisa-Experiments, and RoboTwin2.0 show improvements of STRIKE over the corresponding video-backbone baselines in benchmark measures of physical consistency and manipulation-video fidelity. These results support learned visual state transitions as an effective intermediate representation for physical world modeling.