Driving · 2026-10-08
Kuration SDK: Addressing the Virtual2Real Gap via Data Curation
Nirmit Desai, Eric Song, Mayank Sengupta, Tejal Bedmutha, Siri Reddy, Sahiti Dharmavaram, Kunal Sawarkar
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors report a “Virtual2Real gap” for action-conditioned diffusion world models: qualitative playability on CounterStrike gameplay can diverge sharply from metrics like FVD, LPIPS, and JEDi. They attribute the gap to data curation issues and introduce Kuration SDK, an open-source toolkit for curating gameplay/sequential data and measuring diagnostic properties.
Why it matters
The paper argues that existing benchmarks provide limited signal for domain- and task-agnostic action-conditioned world-model training. By using a data curation toolkit, the authors show that two corpora matched on map/action/state distribution can yield world models with very different playability, while still having similar visual-generation metrics—suggesting diagnostics at the data level are needed to identify causes.
Method
- Reproduce DIAMOND on its published CS:GO dataset, then cross-evaluate DIAMOND on partner-captured CS2 gameplay and vice versa to establish domain gaps.
- Train additional DIAMOND models on CS2 with four data curation strategies using Kuration SDK; evaluate checkpoints with FVD, LPIPS, JEDi plus qualitative interactive rollout inspection.
- Use the SDK to measure an action-state consistency property via pixel_shift and apply action-state-consistency selection by up-weighting segments with larger per-action pixel-shift magnitude.
Limitation
Playability judgments are qualitative from interactive rollout inspection (no labeled playability benchmark exists). Raters are the authors and not blinded to strategy, so expectation bias cannot be ruled out; also, the action-state-consistency training schedule and result are qualitative only, and shared-test-set sweeps with confidence intervals were not run.
Abstract (from arXiv)
Benchmarks for measuring the quality of action-conditioned world models are still evolving and shifting away from visual similarity-based metrics to action-semantic and physically-grounded metrics. However, for domain and task-agnostic action-conditioned world model training, existing benchmarks provide a limited signal. By training and evaluating diffusion world models on CounterStrike gameplay data, we confirm that qualitative playability does not correspond with metrics such as FVD, LPIPS, and JEDi. We term this the Virtual2Real gap. We posit that, in lieu of reliable benchmarks, curating raw gameplay data and measuring a variety of diagnostic properties provides a more robust signal to bridge the gap, before the training even begins. We present several curation strategies and a general-purpose kit for physical AI data curation called Kuration SDK, which is being open-sourced with this paper. The SDK was instrumental in uncovering the root cause of the virtual2real gap in a specific case: why two world models trained on identical gameplay map, action and state distribution, behaved very differently when played in spite of having very similar LPIPS and FVD scores. Thus, Kuration SDK has the potential to uncover the root causes of Virtual2Real gap in specific datasets and accelerate development of sample-efficient training datasets.
Related papers
- LeCuration: A Tiny World Model as a Data Curation Multi-Tool
- Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?
- Scaling in Games: Continued Pre-Training For Embodied Agents In Diverse Virtual Worlds
- PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence
- Player-1: Learning to Play from Synthetic Experience
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC
Kuration SDK: Addressing the Virtual2Real Gap via Data Curation
Preprint posted: 2026-10-07 · Original record ↗
- arxiv
- 2610.09305
Explicit source versions (1)
- v1 · 2026-10-07
- Preprint available
Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Benchmarks for measuring the quality of action-conditioned world models are still evolving and shifting away from visual similarity-based metrics to action-semantic and physically-grounded metrics. However, for domain and task-agnostic action-conditioned world model training, existing benchmarks provide a limited signal. By training and evaluating diffusion world models on CounterStrike gameplay data, we confirm that qualitative playability does not correspond with metrics such as FVD, LPIPS, and JEDi. We term this the Virtual2Real gap. We posit that, in lieu of reliable benchmarks, curating raw gameplay data and measuring a variety of diagnostic properties provides a more robust signal to bridge the gap, before the training even begins. We present several curation strategies and a general-purpose kit for physical AI data curation called Kuration SDK, which is being open-sourced with this paper. The SDK was instrumental in uncovering the root cause of the virtual2real gap in a specific case: why two world models trained on identical gameplay map, action and state distribution, behaved very differently when played in spite of having very similar LPIPS and FVD scores. Thus, Kuration SDK has the potential to uncover the root causes of Virtual2Real gap in specific datasets and accelerate development of sample-efficient training datasets.