Driving · 2026-10-20
UniState: Unifying Memory and Intervention for Controllable World Simulation
Jaeseok Jeong, Kyungmook Choi, Sohyun Chung, Youngsik Yun, Youngjung Uh
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
UniState is a controllable world simulation framework that unifies spatial memory and user interventions in a shared world-state representation. Users specify object motion via 6-DoF transformations in 3D memory, while independently choosing the camera trajectory for observation. The video model follows the specified object motion and camera path using training/sampling (including Scene-Anchored Joint Guidance) that reduce ambiguity between object and viewpoint effects.
Why it matters
The authors target controllable world simulation where interventions can be specified for physical object changes while observing outcomes from freely chosen viewpoints. By grounding both object motion and viewpoint in a single unified world-state representation (shared coordinates) and using guidance/training strategies to align generated videos with both specified object and camera trajectories, the method aims for more accurate joint control than existing approaches.
Method
- Lift the initial image into a unified 3D spatial memory, then express interventions by applying time-varying 6-DoF rigid transformations to selected regions’ geometry inside the same world-state latents.
- Render the intervened world-state along an independently specified camera trajectory to provide camera- and intervention-conditioned inputs for video generation.
- Use Scene-Anchored Joint Guidance: render the initial state under the target camera without interventions to anchor camera-induced background changes, and route/blend velocities to better follow both object motion and camera trajectory.
Abstract (from arXiv)
World models offer increasingly rich environments for exploration, yet intervening in a world while observing the consequences from freely chosen viewpoints remains challenging. We tackle these challenges by introducing UniState, a framework for controllable world simulation built on a unified world-state representation. This representation unifies spatial memory and user interventions. Users specify object motion directly within spatial memory, while independently choosing the camera trajectory from which the resulting world state is observed. This unification makes the relationship between the scene and the requested changes explicit, removing ambiguity between object motion and viewpoint change from the control specification. Crucially, specifying an intervention does not require prescribing every resulting movement: the video model can synthesize additional motion within controlled objects and plausible interactions with their surroundings. We develop training and sampling strategies that help generated videos follow the specified object motion and camera trajectory while preserving this capacity to generate responses beyond the prescribed transformations. Experiments demonstrate more accurate joint object and camera control than the existing methods, alongside qualitative evidence that intervention-induced scene changes persist across rollouts.
Related papers
- ActionRoPE: action as coordinates for spatially consistent control in planar-camera game world models
- StateCraft: Augmenting Video World Models with Abstract-State Reasoning
- Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
- World Observer: Joint Actor-Observer Generation for Persistent World Modeling
- Reading, Writing, Sharing, and Interacting with the World through Video
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-18 · Indexed by Pub-AI: 2026-10-20 UTC
UniState: Unifying Memory and Intervention for Controllable World Simulation
Preprint posted: 2026-10-18 · Original record ↗
- arxiv
- 2610.25414
Explicit source versions (1)
- v1 · 2026-10-18
- Preprint available
Source date: 2026-10-18 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World models offer increasingly rich environments for exploration, yet intervening in a world while observing the consequences from freely chosen viewpoints remains challenging. We tackle these challenges by introducing UniState, a framework for controllable world simulation built on a unified world-state representation. This representation unifies spatial memory and user interventions. Users specify object motion directly within spatial memory, while independently choosing the camera trajectory from which the resulting world state is observed. This unification makes the relationship between the scene and the requested changes explicit, removing ambiguity between object motion and viewpoint change from the control specification. Crucially, specifying an intervention does not require prescribing every resulting movement: the video model can synthesize additional motion within controlled objects and plausible interactions with their surroundings. We develop training and sampling strategies that help generated videos follow the specified object motion and camera trajectory while preserving this capacity to generate responses beyond the prescribed transformations. Experiments demonstrate more accurate joint object and camera control than the existing methods, alongside qualitative evidence that intervention-induced scene changes persist across rollouts.