Video World Models · 2026-11-03

Isekai3D SDG: A Platform for Diverse Synthetic Gameplay Visual Data

Rafal Karp, Gene Liu, Shih-Hsin Li, Eric Courtemanche, Shengze Wang, Josef Spjut, David Pankratz, Mischa Kolbe, Michael Stengel, Sven Elflein, Adam Grygielski, Liwen Wu, Anjul Patney

arXiv:2611.03626PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Isekai3D SDG is a two-stage synthetic data generation platform that separates world simulation/recording from capture. It records world state, character actions, and camera tracks, then replays the same recorded gameplay to produce multiple aligned visual outputs (e.g., RGB, depth, normals) with frame-aligned metadata at chosen resolutions and frame rates. It provides reusable assets and Unreal Engine plugins and a packaged application for data generation without installing the Unreal Engine editor.

Why it matters

The platform targets training data needs for vision/world models and gameplay agents by supporting variation in environments, characters, behaviors, and viewpoints while keeping deterministic alignment between visual sequences and accompanying camera/character/action metadata. Replaying recorded simulation enables multiple output variants from the same underlying activity and supports automated runs and training-friendly lazy datasets over stored videos and metadata.

Method

  • Two-stage workflow: record gameplay (world state, actions, camera trajectories) into Unreal replay + JSON, then replay the recorded sequence to capture selected outputs under configurable rendering/camera settings.
  • Third-person data setup described: 500+ SMPL-X/grounded humanoids via IsekaiCharacters (55-joint layout) and 245+ Fab-sourced environments; configurable behaviors/actions for simulation.
  • Capture outputs include RGB/video (H.264 MP4), optional path-traced H.264, and lossless EXR radiance passes; outputs are stored with frame-aligned camera/character/action metadata and can be read via lazy datasets using object-store range reads.
Abstract (from arXiv)

We present Isekai3D Synthetic Data Generation (SDG), a platform for simulating and capturing synthetic 3D gameplay for research in vision and world models, neural graphics, gameplay agents, and other applications. The platform combines reusable environment and character assets with Unreal Engine plugins in a two-stage workflow. First, world simulation generates gameplay while recording the world state, character movements, actions, and camera tracks needed for replay. Second, the recorded simulation is replayed to capture selected outputs, such as rendered video, images, and depth maps, at a specified resolution and frame rate under selected rendering and camera configurations. Separating simulation from capture allows the same recorded gameplay to produce multiple output variants without repeating the original simulation. Visual outputs are accompanied by frame-aligned camera, character, and action metadata. The platform uploads generated data directly in a training-friendly format, reducing intermediate storage, preparation time, and manual processing. This report describes the design of our overall system, including the platform's assets, plugins, and their functionality, explaining how they work together to generate diverse synthetic gameplay data.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-02 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

Isekai3D SDG: A Platform for Diverse Synthetic Gameplay Visual Data

Rafal Karp, Gene Liu, Shih-Hsin Li, Eric Courtemanche, Shengze Wang, Josef Spjut, David Pankratz, Mischa Kolbe, Michael Stengel, Sven Elflein, Adam Grygielski, Liwen Wu, Anjul Patney

Preprint posted: 2026-11-02 · Original record ↗

arxiv
2611.03626
Explicit source versions (1)
  • v1 · 2026-11-02
  1. Preprint available

    Source date: 2026-11-02 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

We present Isekai3D Synthetic Data Generation (SDG), a platform for simulating and capturing synthetic 3D gameplay for research in vision and world models, neural graphics, gameplay agents, and other applications. The platform combines reusable environment and character assets with Unreal Engine plugins in a two-stage workflow. First, world simulation generates gameplay while recording the world state, character movements, actions, and camera tracks needed for replay. Second, the recorded simulation is replayed to capture selected outputs, such as rendered video, images, and depth maps, at a specified resolution and frame rate under selected rendering and camera configurations. Separating simulation from capture allows the same recorded gameplay to produce multiple output variants without repeating the original simulation. Visual outputs are accompanied by frame-aligned camera, character, and action metadata. The platform uploads generated data directly in a training-friendly format, reducing intermediate storage, preparation time, and manual processing. This report describes the design of our overall system, including the platform's assets, plugins, and their functionality, explaining how they work together to generate diverse synthetic gameplay data.