Video World Models · 2026-11-03
FORCE: Factorized Object Relations for Counterfactual Editing in Video World Models
Suraj K. Sood, Yugyung Lee
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
FORCE is a video world model for counterfactual editing that infers an explicit latent for each ordered object pair from a short observation window without supervision, then holds that relation latent fixed for the whole rollout. Edits directly overwrite the coefficients used by the interaction head to compute forces, enabling removing/changing/transferring/introducing relations. The authors evaluate against simulator counterfactual changes and find worse editing from video than from exact states due to perception precision.
Why it matters
The authors target interrogating world models: given a video-forecasting model, they want edits to persist as relation changes and to evaluate edits by how the outcome changes relative to a simulator under the same operation. Their effect-based scoring separates “did the edit cause the right change?” from ordinary rollout error, and their analysis attributes the video-to-state editing gap to object localization/velocity estimation precision.
Method
- Infer entities with slot attention over frozen DINOv2 features; estimate object positions from attention centroids and velocities from optical flow pooled under each slot’s attention.
- For every ordered object pair, a relation encoder maps pairwise motion over an observation window to a continuous relation latent m_ij, symmetrized and held fixed for the entire rollout.
- An interaction head turns (m_ij) into interaction coefficients (α_ij, β_ij) used to compute equal-and-opposite central forces; counterfactual edits overwrite these coefficients and the model rolls forward with its fixed integrator.
Limitation
Holding m_ij fixed assumes relations persist; the authors state this explicitly and return to it in the limitations.
Abstract (from arXiv)
World models predict how scenes evolve, but they are difficult to interrogate: given a model that forecasts a video, one cannot easily ask what would have happened had a particular interaction between two objects been different. We present Factorized Object Relations for Counterfactual Editing (FORCE), a world model that infers pairwise interactions from raw video without supervision and represents each as an explicit latent per object pair, so that a relation can be removed, changed, transferred or introduced directly. Objects are extracted by slot attention over frozen self-supervised features; a relation encoder infers each pair latent from a short observation window, and an interaction head turns it into a force. The latent is held fixed across the rollout, so an edit persists rather than being recomputed, and every edit acts on the model's own quantities with no mapping to simulator parameters. We evaluate in a simulated environment where the true outcome of an edit is available, scoring the change each edit induces against the change the simulator produces under the same operation. On held-out clips, FORCE's edits made from video agree with the simulator's response at 0.19 to 0.37 on our effect-based score; trained and scored on exact object positions instead, the same architecture reaches 0.57 to 0.85. The gap therefore lies not in the learned relational dynamics but in perception: what bounds an edit is how precisely objects can be located from video, and the precision editing needs lies just beyond what current object-centric perception delivers. Closing that gap is a concrete target for making counterfactual world models work from video.
Related papers
- OnlyAct: Extracting Action Representations from Foundation Models for World Models
- Reading, Writing, Sharing, and Interacting with the World through Video
- Rethinking Video Autoencoders for Interactive World Models
- World-as-Graph: Relational World Modeling Through Latent Space Graphs
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC
FORCE: Factorized Object Relations for Counterfactual Editing in Video World Models
First announced by arXiv: 2026-11-03 · Original record ↗
- arxiv
- 2611.00023
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.00023v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
World models predict how scenes evolve, but they are difficult to interrogate: given a model that forecasts a video, one cannot easily ask what would have happened had a particular interaction between two objects been different. We present Factorized Object Relations for Counterfactual Editing (FORCE), a world model that infers pairwise interactions from raw video without supervision and represents each as an explicit latent per object pair, so that a relation can be removed, changed, transferred or introduced directly. Objects are extracted by slot attention over frozen self-supervised features; a relation encoder infers each pair latent from a short observation window, and an interaction head turns it into a force. The latent is held fixed across the rollout, so an edit persists rather than being recomputed, and every edit acts on the model's own quantities with no mapping to simulator parameters. We evaluate in a simulated environment where the true outcome of an edit is available, scoring the change each edit induces against the change the simulator produces under the same operation. On held-out clips, FORCE's edits made from video agree with the simulator's response at 0.19 to 0.37 on our effect-based score; trained and scored on exact object positions instead, the same architecture reaches 0.57 to 0.85. The gap therefore lies not in the learned relational dynamics but in perception: what bounds an edit is how precisely objects can be located from video, and the precision editing needs lies just beyond what current object-centric perception delivers. Closing that gap is a concrete target for making counterfactual world models work from video.
