Driving · 2026-10-02

Bootstrapping Video Interaction Generation with Synthetic State Transitions

Jiho Jang, Jinyoung Kim, Nojun Kwak, Kyungjune Kim

arXiv:2610.01039PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors introduce a pipeline to bootstrap video interaction generation using synthetic state transitions. They create 'start' and 'end' state images via a taxonomy-guided prompt and state-of-the-art image editing, then generate a coherent video using State-Guided Sampling (SGS) to reduce artifacts. They also propose an automated evaluation system aligned with human judgments. Fine-tuning a base video model on their dataset improves plausibility and interaction quality.

Why it matters

Plausible physical interactions and state transitions are a bottleneck for robotics/VR-like world models. The paper targets this by making interaction outcomes explicit through visual start/end anchors and by sampling trajectories to balance global end-state guidance with local temporal consistency. It also contributes an evaluation framework meant to scale quality assessment using human-aligned automated features.

Method

  • Construct prompts from a custom taxonomy (about 1,300 objects and 500 interaction types) and use GPT-4o to generate interaction state changes and prompts.
  • Generate paired 'start' and 'end' images: use GPT-4o to create an initial image, then use a state-of-the-art image editing model to apply the state change.
  • Generate the video with State-Guided Sampling (SGS), blending velocity fields from an I2V model and a frame infilling/FLF model via dynamic frame-wise weighting to mitigate artifacts; train/evaluate with an automated interaction-quality scoring pipeline.
Abstract (from arXiv)

While recent video generative models can synthesize high-fidelity videos, they struggle to portray plausible physical interactions and the resulting state transitions, a critical bottleneck for applications in robotics and VR/AR. To address this, we introduce a framework to generate a scalable synthetic dataset of controllable interactions. Our pipeline leverages a structured taxonomy and state-of-the-art image editing models to create explicit `start' and `end' state images, which serve as visual anchors for the interaction. To generate a seamless video utilizing these anchors, we propose State-Guided Sampling (SGS), a novel sampling technique that mitigates artifacts common in naive conditional generation. Furthermore, we develop and validate a new automated evaluation system that aligns with human judgments to ensure data quality. Experiments show that fine-tuning a base model on our dataset significantly enhances its ability to generate plausible interactions.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-02 UTC

preprint

Bootstrapping Video Interaction Generation with Synthetic State Transitions

Jiho Jang, Jinyoung Kim, Nojun Kwak, Kyungjune Baek

Preprint posted: 2026-10-01 · Original record ↗

arxiv
2610.01039
Explicit source versions (1)
  • v2 · 2026-10-19
  1. Explicit source version detected · v2

    Source date: 2026-10-19 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
  2. Preprint available

    Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

While recent video generative models can synthesize high-fidelity videos, they struggle to portray plausible physical interactions and the resulting state transitions, a critical bottleneck for applications in robotics and VR/AR. To address this, we introduce a framework to generate a scalable synthetic dataset of controllable interactions. Our pipeline leverages a structured taxonomy and state-of-the-art image editing models to create explicit `start' and `end' state images, which serve as visual anchors for the interaction. To generate a seamless video utilizing these anchors, we propose State-Guided Sampling (SGS), a novel sampling technique that mitigates artifacts common in naive conditional generation. Furthermore, we develop and validate a new automated evaluation system that aligns with human judgments to ensure data quality. Experiments show that fine-tuning a base model on our dataset significantly enhances its ability to generate plausible interactions.