Robotics · 2026-10-14
Remembering What Matters for Long-Horizon Robotic World Models
Hao Wu, Tianyu Shao, Shengju Qian, Yizhou Zhao, Fan Zhang, Penghao Zhao, Weiyan Wang, Yufei Huang, Fan Xu, Kun Wang, Xin Wang, Ming-Hsuan Yang, Yuxuan Liang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ReCAP is a training-free inference-time context scheduler for discrete-token autoregressive robot video world models. It keeps the initial scene anchor and a contiguous window of recent visual–action blocks while evicting older generated history, keeping token/KV-cache growth bounded. Using frozen world models, the authors report improved long-horizon visual fidelity, action-conditioned consistency, and robot structure preservation vs Full Context across RT-1, BridgeV2, LIBERO-90, and CALVIN.
Why it matters
The authors address unreliability of discrete-token robot video world models over long rollouts under Full Context decoding, where retaining all generated history can worsen attention to recent states and increase KV/cache costs. ReCAP shows that reorganizing existing inference context (without retraining or architectural changes) can improve prediction quality while reducing memory/latency, offering a practical path for long-horizon robot world modeling.
Method
- Introduce ReCAP, a training-free context-scheduling method that preserves a persistent scene anchor and retains the most recent W complete visual–action blocks while evicting older generated blocks.
- Keep the pretrained tokenizer, world model, and visual decoder frozen; only change the inference prompt/KV-cache selection schedule, operating blockwise in token space.
- Evaluate on RT-1, BridgeV2, LIBERO-90, and CALVIN with frozen autoregressive world models, comparing to Full Context and analyzing attention and context interventions.
Limitation
The authors state that their history-intervention observations do not isolate the memory-role-conflict hypothesis from context-length and position-handling effects.
Abstract (from arXiv)
Autoregressive robot video world models based on discrete visual tokens remain unreliable over long rollout horizons. Retaining the entire generated history does not necessarily improve prediction quality. We investigate a possible memory-role conflict in Full Context decoding: persistent scene information, recent visual-action states, and stale generated history compete for attention despite their different predictive roles. This perspective motivates ReCAP (Receding Context with Anchor Preservation), a training-free context-scheduling method for long-horizon robot video world models. ReCAP keeps the pretrained tokenizer, world model, and visual decoder frozen. It preserves the initial scene anchor and a contiguous window of recent complete visual-action blocks while evicting older generated history. The schedule preserves within-block structure and temporal order, requires no additional modules or parameter updates, and bounds active-context growth independently of rollout length. We evaluate ReCAP on RT-1, BridgeV2, LIBERO-90, and CALVIN using frozen autoregressive world models. Across evaluated environments and horizons, ReCAP improves visual fidelity, action-conditioned consistency, and robot structure preservation relative to Full Context. On BridgeV2, it reduces LPIPS by 63.3%. In the system benchmark at H=64, it reduces the logical KV footprint by 73.3% and p50 latency by 6.51%. Attention analyses show increased attention to retained recent states, while history interventions associate larger amounts of stale context with higher prediction error. These observations are consistent with the memory-role-conflict hypothesis, although they do not isolate it from context-length and position-handling effects. Our results support inference-time context scheduling as a practical approach to improving long-horizon robot video prediction and computational efficiency without retraining.
Related papers
- Long-WAM: Scaling the Context of World-Action Models
- Learning Robotic World Models with Semantic and Visual Feedback
- LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
- FutureWorlds: Learning Robotic World Models from Alternative Futures
- MOST: Memory of Object States and Task Progress for Long-Horizon Mobile Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-10 · Indexed by Pub-AI: 2026-10-14 UTC
Remembering What Matters for Long-Horizon Robotic World Models
Preprint posted: 2026-10-10 · Original record ↗
- arxiv
- 2610.14583
Explicit source versions (1)
- v1 · 2026-10-10
- Preprint available
Source date: 2026-10-10 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Autoregressive robot video world models based on discrete visual tokens remain unreliable over long rollout horizons. Retaining the entire generated history does not necessarily improve prediction quality. We investigate a possible memory-role conflict in Full Context decoding: persistent scene information, recent visual-action states, and stale generated history compete for attention despite their different predictive roles. This perspective motivates ReCAP (Receding Context with Anchor Preservation), a training-free context-scheduling method for long-horizon robot video world models. ReCAP keeps the pretrained tokenizer, world model, and visual decoder frozen. It preserves the initial scene anchor and a contiguous window of recent complete visual-action blocks while evicting older generated history. The schedule preserves within-block structure and temporal order, requires no additional modules or parameter updates, and bounds active-context growth independently of rollout length. We evaluate ReCAP on RT-1, BridgeV2, LIBERO-90, and CALVIN using frozen autoregressive world models. Across evaluated environments and horizons, ReCAP improves visual fidelity, action-conditioned consistency, and robot structure preservation relative to Full Context. On BridgeV2, it reduces LPIPS by 63.3%. In the system benchmark at H=64, it reduces the logical KV footprint by 73.3% and p50 latency by 6.51%. Attention analyses show increased attention to retained recent states, while history interventions associate larger amounts of stale context with higher prediction error. These observations are consistent with the memory-role-conflict hypothesis, although they do not isolate it from context-length and position-handling effects. Our results support inference-time context scheduling as a practical approach to improving long-horizon robot video prediction and computational efficiency without retraining.
