Video World Models

Generative video models used as world simulators. (192 papers)

2026-11-13 · arXiv:2611.15082

Prediction or Persistence? What Future-Prediction Scores Miss About World Models

Animesh Maheshwari

The authors argue that “future-prediction” scores for pretrained video world models mix encoder quality, near-future similarity to the recent past, and the predictor’s own contribution. They add a matched persistence control (predictor replaced by identity) and show predictor contribution can be positive, small, or negative even when the original score is high. They recommend reporting future-prediction scores with their persistence baseline.

2026-11-13 · arXiv:2611.15131

Duo: A Reciprocal World Model for Full-Duplex Interaction

Hengyuan Cao, Biao Gong, Guoshan Lu et al.

The authors introduce the Reciprocal World Model (RWM) and Duo v1, a full-duplex world model that continuously perceives real-time audio, video, and text while responding in real time. Duo v1 uses a chunk-wise setup with an Autoregressive Intent Module for participation decisions and a causal audiovisual Realization Module that generates audio/video latents conditioned on intent, synchronized on a unified timeline. They train in stages and distill for live interaction.

Video World ModelsCode availableAI summary
2026-11-13 · arXiv:2611.15143

Multi-View 3D Reconstruction Using V-JEPA Features

Anwar Shamim, Jeffrin Sam

The authors study supervised RGB-only multi-view 3D reconstruction using a downstream architecture initialized from a V-JEPA 2.1 encoder. They compare global vs spatial camera estimation and a joint relative-objective configuration on CO3D (five-view windows) and an independently trained TUM RGB-D setting. Results show mixed gains: spatial cameras improve CO3D absolute trajectory error but hurt trajectory/depth relative to identity/global; relative objectives improve calibrated/aligned-shape metrics but worsen raw reconstruction.

Video World ModelsCode availableAI summary
2026-11-13 · arXiv:2611.15585

Marginal-Preserving Coupled Diffusion Composition for Long-Horizon Planning

Hengrui Zhang, Qiyu Wang, Yuhu Cheng et al.

The authors propose MCDC, a training-free inference modification for pretrained compositional diffusion planners. Instead of sampling chunk initial noises independently, MCDC samples coupled Gaussian initializations with exact per-chunk standard-Gaussian marginals while inducing positive cross-chunk correlation on neighboring chunks, improving overlap-mode compatibility and long-horizon stitching coherence.

2026-11-13 · arXiv:2611.15602

CoCode-World: Shared 3D Proxy Representation for Multi-Player World Models

Qirui Li, Xin Zhang, Yabo Chen et al.

CoCode-World is a shared 3D proxy framework for joint two-player video generation. It uses programmable, textureless 3D proxies for layout/motion/occlusion and an RGB appearance reference for characters and scene style. The two players’ proxy videos and appearance evidence are concatenated into one context, and a pretrained video model is adapted with LoRA so native attention enables cross-view communication and consistent player identity.

2026-11-13 · arXiv:2611.16217

WorldDraft: If Code Builds the World, Can Video Models Just Render It?

Zhiyu Hou, Wen-Jie Shu, Harold Haodong Chen et al.

WorldDraft studies whether video models can act as draft-to-video generative renderers when 4D world dynamics and camera paths are provided by executable code. It introduces a benchmark (432 strictly paired cases across 48 4D world families) to jointly measure world preservation and visual realization over varying draft granularity and appearance alignment. The authors propose WorldDraft-Primer to prompt frozen video renderers for better adherence.

Video World ModelsCode availableAI summary
2026-11-13 · arXiv:2611.16221

BWM-Forge: Agentic World Programming for Embodied Intelligence

BWM Team

BWM-Forge is a code-driven framework for controllable robot video generation. It uses a coding agent to build editable Blender scene programs specifying objects, robot motion, and camera states, then renders proxy videos that provide structural and motion references for a video generation model that adds visual detail. A multi-agent harness (preprocessing, building, review, rendering) supports iterative refinement and experience reuse across tasks, robot embodiments, and viewpoints.

Video World ModelsCode availableAI summary
2026-11-11 · arXiv:2611.13818

WAM is capable in BEHAVIOR-1K: the 2nd-place solution to the 2026 BEHAVIOR Challenge

Yelin He, Jiaquan Ye, Xinming Wei et al.

The authors present the 2nd-place Team IF1001 solution to the 2026 BEHAVIOR Challenge (mobile manipulation over 100 household tasks). The method is built on Cosmos 3 as a world action model (WAM) and improved with richer visual history, task-progress estimation, task-group and skill-specific LoRA adapters, plus inference-time routing and execution strategies. The system achieves 0.615 mean Q-score and 41.2% full-task success on the official private test set.

2026-11-11 · arXiv:2611.14420

OnlyAct: Extracting Action Representations from Foundation Models for World Models

Runjia Li, Minghao Chen, Junyu Xie et al.

OnlyAct extracts action representations from a pretrained video foundation model (V-JEPA 2) using label-free contrastive adaptation. It constructs positives by keeping motion while swapping scenes (flow-guided cut-and-paste) and uses hard negatives from the same video with different motion. The adapted encoder conditions an action-conditioned autoregressive video world model, enabling action transfer and real-action control on robot, camera-navigation, game, and driving benchmarks.

2026-11-10 · arXiv:2611.10013

From Video to Digital Twin: An Exploratory Study of World Models and Gaussian Splatting on In-the-Wild Video

Somesh Mohapatra

The authors study generative world models and locally optimized 3D Gaussian splats on an uncurated in-the-wild news walkthrough of a food bank warehouse (20 frames for a commercial world model). The world model recovered camera poses and metric depth for three areas and fused them into ~747k Gaussians. Held-out novel-view PSNR drops when training uses only 8 frames, and locally trained splats degrade quickly away from the filmed path. Generating unseen views failed on the provider side twice.

2026-11-10 · arXiv:2611.10392

Tiresias: Towards Language-Conditioned World Models

Fengyi Wu, Yifei Dong, Yetong Sha et al.

The authors introduce LCWM, a language-conditioned world modeling task that maps an initial agent-view observation and an instruction to an action sequence fulfilling the instruction, and its benchmark LCWM-Bench spanning 15 datasets across navigation, robotics, 3D gaming, and driving. They propose Tiresias, which plans via selective imagination using a diffusion policy for trajectory proposals, trajectory/visual rankers, and a video-pretrained world model for imagining consequences.

Video World ModelsCode availableAI summary
2026-11-10 · arXiv:2611.11210

DynOLAT: OLAT Video Prediction for Controllable Dynamic Full-Body Relighting

Taoming Liu, Yinji ShenTu, Guangzhao He et al.

DynOLAT is an OLAT video prediction framework for controllable dynamic full-body relighting. It predicts temporally consistent One-Light-At-a-Time (OLAT) basis videos from flat-lit inputs, then linearly combines predicted bases using weights from a target environment map. The authors introduce a dynamic capture pipeline and a large-scale dynamic full-body OLAT dataset (over 24M frames) and report improved lighting-response fidelity and temporal consistency versus tested baselines.

2026-11-10 · arXiv:2611.11500

ForeAct: Geometry-Guided Action Learning from World Representations

Wei Lu, Jiaxi Zhang, Yunheng Wang et al.

ForeAct repurposes a pretrained video world model as a single-pass, low-latency backbone for robotic manipulation. It processes the clean observation latent in one forward pass (no future-frame synthesis or iterative denoising) and introduces learnable geometry tokens supervised by next-frame relative depth via bidirectional self-attention. A lightweight geometry readout is used only during training and removed at deployment.

2026-11-10 · arXiv:2611.11697

SurgTAP-3D: A Large-Scale Cross-Surgery Video Dataset with Contact-Aware 3D Motion Tracking

Jiaxin Guo, Wenzhen Dong, Taoyu Wu et al.

SurgTAP-3D is a large-scale in-vivo pseudo-3D surgical motion tracking dataset and benchmark for contact-aware 3D point tracking in minimally invasive surgery. It includes 520K frames, 4,549 action clips, and 1.4M 2D/3D point trajectories from diverse robotic and manual procedures under unified action categories. The authors propose a unified framework to convert uncalibrated monocular surgical videos into structured 4D annotations via contact-region prompting, active-tool segmentation, dynamic bundle adjustment, and global 4D refinement.

2026-11-10 · arXiv:2611.12319

Mitigating SM-Level Load Imbalance in 3D Gaussian Splatting on GPUs

Junha Ryu, Hoi-Jun Yoo

The authors identify SM-level load imbalance in GPU-based 3D Gaussian Splatting (3DGS) alpha blending caused by uneven per-tile workloads, which creates a long execution tail. They propose software scheduling: persistent-kernel LPT dispatch of heavy tiles and selective depth-wise chunking with saturation signaling. A lightweight per-frame policy decides chunking. They report alpha blending speedups up to 2.93× averaged on a large-scale urban scene and up to 5.6× on the most imbalanced frames.

2026-11-10 · arXiv:2611.12353

QCC-WM: Linking Macro-State Dynamics and Visual Generation via Quotient Causal Construction

Zhizhong Fu, Wei Zhou, Zhaoyang Jiang et al.

QCC-WM is an action-conditioned visual world model that constructs macro-states as quotient-based causal macro-nodes. Action responses guide macro-variable construction and a shared reference quotient measurement constrains consistency between predicted macro-states and observed changes. Macro predictions drive motion-guided reference transport and readback with node-level error correction for future visual generation. On US-VLA, it reports improved ROI-MSE/LPIPS for multi-step ultrasound video prediction.

2026-11-10 · arXiv:2611.12557

WanderBench: Can World Models Explore Dynamic Worlds from Video?

Yuhang Chen, Jiheng Li, Jiale Wang et al.

The authors introduce WanderBench, a benchmark for exploring dynamic worlds from a recorded source video with camera routes given as keyboard actions. Using 745 single-shot clips and 24 sub-metrics over six dimensions, they report that longer horizons degrade camera control and source-dynamics fidelity and that no model keeps revisited places consistent. They propose WanderLand, a training-free conditioning framework, reporting gains up to +8.3 points in source-dynamics fidelity.

2026-11-09 · arXiv:2611.09621

BiLoGo: Reducing the smoothing of localised responses in Transformer-based neural operators for sheet forming

Yingxue Zhao, Haoran Li, Haosu Zhou et al.

BiLoGo is an FE-guided learned simulator for sheet forming that reduces smoothing of localized thinning responses. It combines bidirectional node–element GraphSAGE (no interpolation), a dual-stream Transformer neural operator with separate tokens for blank deformation and tool contact, and gated residual fusion with element updates conditioned on element-node kinematics. The authors evaluate on dome and two corner cases and report lowest positional and threshold-based thinning errors, with higher rib-near smoothing index than token-only Transformer models.

2026-11-09 · arXiv:2611.09894

Video-Coder: Reinforcing Agentic Coding for Video Generation

Chaoyang Wang, Sicheng Gao, Junan Hu et al.

Video-Coder is a code-driven video generation agent. It writes and executes Blender Python code to build a dynamic 3D scene and renders a “Draft Video” that provides structure and motion signals for a fixed downstream video generator. The method trains with SFT on Video-Coder-SFT-6k and then harness-native agentic RL on Video-Coder-RL-2k using execution- and video-based rewards with an annealing schedule.

Video World ModelsCode availableAI summary
2026-11-06 · arXiv:2611.07245

MAC-v2: Long-horizon model-based value expansion from pixels

Kwanyoung Park, Sergey Levine

MAC-v2 is an offline model-based RL method for pixel observations that enables long-horizon model-based value expansion. The authors train a reconstruction-free latent world model with action chunking, learning representations from prediction and control objectives. This reduces recursive compounding of prediction error and allows long-horizon latent imagination over visual benchmarks (visual OGBench, MMBench2) and robotics tasks (Robomimic, LIBERO-10).