Video World Models · 2026-10-14
World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
Junyi Wu, Zekun Li, Chenhongyi Yang, Chen Guo, Lingchen Yang, Yongan Zhang, Xinyi Zhang, Hongbing Hu, Bartek Rymkowski, Srinath Sridhar, Yan Yan, Xijie Huang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
World-T3 is a camera-controlled video world model that uses test-time training (TTT) as persistent, pose-indexed memory. During long-horizon generation, evicted blocks are consolidated into fast-weight parameters via a recursive least-squares rule, and later reads retrieve prior scene features by camera pose. The autoregressive model is distilled into a 4-step generator for real-time interactive streaming.
Why it matters
The authors target long-horizon revisit consistency in video world models by preventing spatial memory decay when the generated context window no longer contains earlier views. World-T3 aims to consolidate past observations into fixed-size parametric memory (constant storage and per-step computation), while still supporting real-time generation via few-step distillation.
Method
- Stage 2 integrates a test-time training (TTT) memory into a block-causal autoregressive video diffusion model, reading memory by target camera pose and writing evicted context blocks into fast-weight parameters.
- Memory writes are robustified with a recursive least-squares (ridge regression) update over pose-key/value pairs to reduce redundancy from smooth camera trajectories and make the consolidated memory order-invariant.
- Stage 3 distills the autoregressive model into a 4-step generator using dual-teacher distribution matching, combining a bidirectional teacher (local quality) and a memory-augmented autoregressive teacher (long-horizon consistency).
Limitation
The framework relies on 6-DoF camera trajectories extracted by 3D foundation models (e.g., VGGT-Ω) for conditioning; off-the-shelf pose estimators can exhibit tracking noise or subtle drift in challenging environments, and since TTT memory consolidation is pose-keyed, such inaccuracies can degrade query-to-memory alignment and accumulate over extended horizons.
Abstract (from arXiv)
Recent advances in video world models have enabled plausible interactive environments, yet long-horizon generation places a fundamental demand on visual memory: the previously observed world must remain consistent as the model continues to generate. Existing autoregressive approaches often implicitly rely on the generative model itself to retain such information, leading to spatial memory decay beyond the active context. We introduce World-T3, a camera-controlled video world model with a test-time training memory that consolidates past observations into dedicated model parameters. To robustly update this memory, we propose a recursive least-squares rule that incrementally absorbs novel visual features via online pose-keyed memory updates. Our TTT memory reads out previously observed scene features from the fast-weight state by camera pose, enabling consistent generation with constant storage and per-step computation regardless of rollout length. To facilitate real-time interactive generation with this robust memory, we further distill the autoregressive model into a few-step generator via dual-teacher distribution matching. On a long-horizon revisit benchmark, World-T3 raises revisit PSNR from 13.52 to 18.31 dB and reduces absolute rotation error from 3.344 to 1.075 relative to the strongest baseline on both metrics, while sustaining 20.6 FPS at 1280 x 704 on a single NVIDIA B200 GPU.
Related papers
- WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models
- Persistent Planar Memory for Video World Models
- Momentum: Persistent Dynamics for Video World Models
- Learning to Retrieve: Internalizing Memory Retrieval for Video World Models
- Honeycomb: Constant-Size Scene Memory Representation for Video World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-12 · Indexed by Pub-AI: 2026-10-14 UTC
World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
Preprint posted: 2026-10-12 · Original record ↗
- arxiv
- 2610.17434
Explicit source versions (1)
- v1 · 2026-10-12
- Preprint available
Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent advances in video world models have enabled plausible interactive environments, yet long-horizon generation places a fundamental demand on visual memory: the previously observed world must remain consistent as the model continues to generate. Existing autoregressive approaches often implicitly rely on the generative model itself to retain such information, leading to spatial memory decay beyond the active context. We introduce World-T3, a camera-controlled video world model with a test-time training memory that consolidates past observations into dedicated model parameters. To robustly update this memory, we propose a recursive least-squares rule that incrementally absorbs novel visual features via online pose-keyed memory updates. Our TTT memory reads out previously observed scene features from the fast-weight state by camera pose, enabling consistent generation with constant storage and per-step computation regardless of rollout length. To facilitate real-time interactive generation with this robust memory, we further distill the autoregressive model into a few-step generator via dual-teacher distribution matching. On a long-horizon revisit benchmark, World-T3 raises revisit PSNR from 13.52 to 18.31 dB and reduces absolute rotation error from 3.344 to 1.075 relative to the strongest baseline on both metrics, while sustaining 20.6 FPS at 1280 x 704 on a single NVIDIA B200 GPU.