Driving · 2026-10-02
Bootstrapping Video Interaction Generation with Synthetic State Transitions
Jiho Jang, Jinyoung Kim, Nojun Kwak, Kyungjune Kim
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce a pipeline to bootstrap video interaction generation using synthetic state transitions. They create 'start' and 'end' state images via a taxonomy-guided prompt and state-of-the-art image editing, then generate a coherent video using State-Guided Sampling (SGS) to reduce artifacts. They also propose an automated evaluation system aligned with human judgments. Fine-tuning a base video model on their dataset improves plausibility and interaction quality.
Why it matters
Plausible physical interactions and state transitions are a bottleneck for robotics/VR-like world models. The paper targets this by making interaction outcomes explicit through visual start/end anchors and by sampling trajectories to balance global end-state guidance with local temporal consistency. It also contributes an evaluation framework meant to scale quality assessment using human-aligned automated features.
Method
- Construct prompts from a custom taxonomy (about 1,300 objects and 500 interaction types) and use GPT-4o to generate interaction state changes and prompts.
- Generate paired 'start' and 'end' images: use GPT-4o to create an initial image, then use a state-of-the-art image editing model to apply the state change.
- Generate the video with State-Guided Sampling (SGS), blending velocity fields from an I2V model and a frame infilling/FLF model via dynamic frame-wise weighting to mitigate artifacts; train/evaluate with an automated interaction-quality scoring pipeline.
Abstract (from arXiv)
While recent video generative models can synthesize high-fidelity videos, they struggle to portray plausible physical interactions and the resulting state transitions, a critical bottleneck for applications in robotics and VR/AR. To address this, we introduce a framework to generate a scalable synthetic dataset of controllable interactions. Our pipeline leverages a structured taxonomy and state-of-the-art image editing models to create explicit `start' and `end' state images, which serve as visual anchors for the interaction. To generate a seamless video utilizing these anchors, we propose State-Guided Sampling (SGS), a novel sampling technique that mitigates artifacts common in naive conditional generation. Furthermore, we develop and validate a new automated evaluation system that aligns with human judgments to ensure data quality. Experiments show that fine-tuning a base model on our dataset significantly enhances its ability to generate plausible interactions.
Related papers
- BoxWorld: Let World Engines Control and Video Generators Render
- ReactWorld: Structuring Interaction Responses for Embodied Video Generation
- PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements
- StateCraft: Augmenting Video World Models with Abstract-State Reasoning
- TAPESTRY: A Single Backbone for Video Generation and Robot Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC
Bootstrapping Video Interaction Generation with Synthetic State Transitions
Preprint posted: 2026-10-01 · Original record ↗
- arxiv
- 2610.01039
Explicit source versions (1)
- v2 · 2026-10-19
- Explicit source version detected · v2
Source date: 2026-10-19 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Preprint available
Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
While recent video generative models can synthesize high-fidelity videos, they struggle to portray plausible physical interactions and the resulting state transitions, a critical bottleneck for applications in robotics and VR/AR. To address this, we introduce a framework to generate a scalable synthetic dataset of controllable interactions. Our pipeline leverages a structured taxonomy and state-of-the-art image editing models to create explicit `start' and `end' state images, which serve as visual anchors for the interaction. To generate a seamless video utilizing these anchors, we propose State-Guided Sampling (SGS), a novel sampling technique that mitigates artifacts common in naive conditional generation. Furthermore, we develop and validate a new automated evaluation system that aligns with human judgments to ensure data quality. Experiments show that fine-tuning a base model on our dataset significantly enhances its ability to generate plausible interactions.
