Video World Models · 2026-10-22

Rethinking Video Autoencoders for Interactive World Models

Adrien Ramanana Rahary, Yutong Bai, Nicolas Dufour, Patrick P\'erez, David Picard

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors study how video-world-model compression affects interactive latency/throughput and rollout/action quality. They introduce CADRE, an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions. Training the same causal diffusion world model on each compression, CADRE “forms most of the latency-quality frontier,” especially with a moving camera, and yields findings about where/how much to compress and diffusion vs perceptual regression decoding.

Why it matters

Interactive world models must trade latency against rollout quality and action faithfulness. Prior work often reuses offline text-to-video autoencoders, inheriting compression choices misaligned with agent waiting constraints. CADRE isolates how spatial vs temporal compression (and decoder choice) affect latency-quality/action-following behavior, with reported real-time throughput/latency measurements and a downloadable demo/models link.

Method

Build CADRE: multiple causal diffusion representation autoencoders that differ only in a small bottleneck on frozen per-frame features, sharing a causal diffusion decoder self-distilled to one or a few decoding steps.
Train the same 1.2B causal diffusion transformer world model (conditioned on τ actions per latent frame) on each autoencoder’s latents, across manipulation and navigation environments.
Evaluate each autoencoder via rollout image/motion quality (FID, FVD), reconstruction-style metrics (PSNR, LPIPS), and action following via action recoverability ratio (ARR), while measuring action lag/latency from decoding and chunking.

Limitation

The paper does not state a specific limitation in the provided text beyond noting metric fragility (e.g., FID) and evaluation protocol choices; no direct limitation statement is provided.

Abstract (from arXiv)

Video world models predict future frames from past frames and actions. They serve as simulators in which agents train, plan or are evaluated, and that humans can control interactively. They generate in the latent space of a video autoencoder, whose compression sets their cost per frame and how long each action waits to take effect. Yet world models often reuse the autoencoders of text-to-video generators, built for offline generation with no agent waiting and, without their training data and code, not retrainable at another compression. We therefore ask how, where and how much a world model should compress. We build CADRE, an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions. Its members differ only in a small bottleneck on frozen features and share a diffusion decoder self-distilled to one or a few steps. We train the same world model on each autoencoder in two domains, manipulation and navigation, with two environments each. CADRE forms most of the latency-quality frontier, especially with a moving camera. Our study also leads to four findings. At strong compression, a diffusion decoder is sharper than perceptual regression. In robot manipulation, temporal compression is more effective in the autoencoder than in the world model. Manipulation and navigation call for opposite compressions: temporal compression costs little when the camera is still, and spatial compression when it moves. Surprisingly, with a still camera, coarser spatial tokens hurt motion far more than image quality. Code, models and a live demo are at https://kyutai.org/blog/2026-10-21-cadre

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC

preprint