Video World Models · 2026-10-19
Execution Before Memory: Diagnosing Memory Failures in Video World Models
Weiyang Jin, Jiwen Yu, Sihan Xu, Yuwei Niu, Xihui Liu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors argue that video world-model “memory” tests should separate (1) scene setup, (2) whether the hidden change leaks early, and (3) whether the final reveal actually shows the change. They introduce MIST to score these gates separately, showing failures often come from missing execution of the instructed change rather than forgetting. They propose M3-Agent (test-time instruction revision) and report improvements on WRBench and MBench-T.
Why it matters
A single low memory score can conflate forgetting with failures to execute the asked change; the authors show this ambiguity and redesign evaluation to localize where failures occur (scene setup vs execution vs reveal). Their M3-Agent shows that the same frozen generator can perform missing moves once the failed operation is revised, clarifying what “memory” scores are actually measuring.
Method
- Introduce MIST to score delayed-reveal memory by separate gates: setup, gap leakage, and final reveal/inside match; missing execution cannot be mistaken for forgetting when the action never becomes visible.
- Diagnose failures by inspecting the last frame and classifying failure modes (e.g., no action, wrong binding, missing departure/return, return to intact object, occluded view).
- Propose M3-Agent: at test time, rewrite only the failed operation/instruction details (not the answer) and regenerate with the same frozen generator; keep candidates selected by the score tied to the relevant evaluation.
Limitation
The authors report that a judged last frame does not show stored memory: “They do show that the model can already carry out many missing changes once the instruction names them. A judged last frame is not stored memory (§ 8 ).”
Abstract (from arXiv)
A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We introduce MIST, an evaluation that separates the change from the memory it should leave behind. The model first sets the scene while the change stays hidden, and only later is asked to show it, with leakage before that point kept under control. This separation locates the bottleneck: the model sets the scene but does not carry out the instructed change. The same gap appears in general memory evaluations. On that basis we propose M3-Agent (Make the Missing Move), which recursively revises the details of the input instruction so that the model successfully carries out the requested instruction. Relative to the baseline, it more than triples the MIST reveal score. On the benchmarks, WRBench's score T rises by 26.1% and MBench-T's score U by 52.7%. The method meets the problem from the execution side: it shows what the frozen model can do once the move is stated, and it sets a stricter baseline.
Related papers
- WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models
- World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
- Learning to Retrieve: Internalizing Memory Retrieval for Video World Models
- MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
- MemoWM: How World Models Change What Agents Need to Remember
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-16 · Indexed by Pub-AI: 2026-10-19 UTC
Execution Before Memory: Diagnosing Memory Failures in Video World Models
Preprint posted: 2026-10-16 · Original record ↗
- arxiv
- 2610.23542
Explicit source versions (1)
- v1 · 2026-10-16
- Preprint available
Source date: 2026-10-16 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We introduce MIST, an evaluation that separates the change from the memory it should leave behind. The model first sets the scene while the change stays hidden, and only later is asked to show it, with leakage before that point kept under control. This separation locates the bottleneck: the model sets the scene but does not carry out the instructed change. The same gap appears in general memory evaluations. On that basis we propose M3-Agent (Make the Missing Move), which recursively revises the details of the input instruction so that the model successfully carries out the requested instruction. Relative to the baseline, it more than triples the MIST reveal score. On the benchmarks, WRBench's score T rises by 26.1% and MBench-T's score U by 52.7%. The method meets the problem from the execution side: it shows what the frozen model can do once the move is stated, and it sets a stricter baseline.