Robotics · 2026-10-28
Rollback: Experience Backtracking in World Models for Policy Improvement
Jiayi Luo, Yang Zhang, Liliang Chen, Nan Wang, Chen Gao, Boxiang Qiu, Zhuoxuan Li, Baorui Peng, Chuankang Li, Yuxiang Yan, Si Liu, Qingyun Sun, Shuicheng Yan, Maoqing Yao, Jianxin Li, Guanghui Ren
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Rollback turns action-conditioned world models into counterfactual engines by “experience backtracking”: it rewinds to critical decision points, branches into alternative action continuations, and distills relative action-quality supervision from contrasted outcomes—without an additional learned value model. A distributed training loop co-evolves the world model with asynchronous real-robot execution via test-time physics grounding (LoRA-based adaptation and distillation).
Why it matters
The authors argue world-model RL often uses forward rollouts with discarded contexts, missing the ability to “rewind and reconsider” past decision points available to learned models. Rollback targets sparse decision-level credit assignment and world-model exploitation by combining experience-backtracking tree search, tree-structured outcome aggregation for action-level supervision, and test-time physics grounding through real transitions.
Method
- Experience-backtracking tree search: roll back to cached intermediate decision contexts and simulate alternative action continuations; MCTS allocates imagination budget toward decisions with divergent outcomes.
- Tree-structured value estimation: subtree outcome aggregation produces chunk-level relative action-quality labels for policy training without a separately learned value model.
- World model–policy co-evolution (test-time physics grounding): while the world model serves as the training environment, real transitions collected by the policy correct predicted dynamics via a lightweight LoRA pipeline.
Limitation
Rollback relies on a task-specific reward model and on world-model fidelity over long contact-rich horizons; the fidelity analysis shows prediction error grows along a rollout. Each policy in this work is trained for a single task with one physical robot.
Abstract (from arXiv)
Action-conditioned world models offer a scalable alternative to costly real-world interaction for improving robot policies. However, existing world-model-based methods largely treat world models as forward-only surrogates of the physical environment, leaving untapped a capability no physical environment affords: rewinding to past decision points to explore counterfactual futures. We introduce Rollback, a policy improvement paradigm that turns world models from passive simulators into counterfactual engines: it rolls back to critical decision points, branches into alternative actions, and contrasts their outcomes to attribute success to individual decisions. This concentrates imagination where it matters most and distills counterfactual outcomes into fine-grained, action-level supervision without a separately learned value model. We build a scalable training system that unifies distributed world-model search with asynchronous real-robot execution, forming a self-reinforcing cycle: real-world experience continually sharpens the world model, and a sharper world model in turn drives further policy improvement. We term the world-model side of this cycle test-time physics grounding, a capability that hand-built physics simulators lack: while the world model serves as the training environment, real transitions from the policy it trains correct its dynamics through a lightweight LoRA pipeline. Experiments on three challenging real-world tasks show that Rollback improves success rates over the strongest baseline by an absolute 10.67%, 11.11%, and 11.85% on block extraction, ball tossing, and tabletop curling, respectively, while achieving more stable improvement throughout training. Although initialized from teleoperated demonstrations far below human level, Rollback surpasses the average success rate of human players on both ball tossing and tabletop curling within 8 hours of training.
Related papers
- Right Ranking, Wrong Scale: Reproducing the Robotic World Model's Dynamics Model and Testing the Uncertainty of Its Follow-up
- World Models Dream of Success: Diagnosing and Repairing Failure Insensitivity in Robot World Models
- pico-WAM: Evaluating Counterfactual Prediction and Planning Error in a Compact World Action Model
- The Planning Limits of Latent World Models
- $R^2$-WAM: Repair-and-Reject Post-Training for World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-26 · Indexed by Pub-AI: 2026-10-28 UTC
Rollback: Experience Backtracking in World Models for Policy Improvement
Preprint posted: 2026-10-26 · Original record ↗
- arxiv
- 2610.36068
Explicit source versions (1)
- v1 · 2026-10-26
- Preprint available
Source date: 2026-10-26 · First detected: 2026-10-28 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.36068v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Action-conditioned world models offer a scalable alternative to costly real-world interaction for improving robot policies. However, existing world-model-based methods largely treat world models as forward-only surrogates of the physical environment, leaving untapped a capability no physical environment affords: rewinding to past decision points to explore counterfactual futures. We introduce Rollback, a policy improvement paradigm that turns world models from passive simulators into counterfactual engines: it rolls back to critical decision points, branches into alternative actions, and contrasts their outcomes to attribute success to individual decisions. This concentrates imagination where it matters most and distills counterfactual outcomes into fine-grained, action-level supervision without a separately learned value model. We build a scalable training system that unifies distributed world-model search with asynchronous real-robot execution, forming a self-reinforcing cycle: real-world experience continually sharpens the world model, and a sharper world model in turn drives further policy improvement. We term the world-model side of this cycle test-time physics grounding, a capability that hand-built physics simulators lack: while the world model serves as the training environment, real transitions from the policy it trains correct its dynamics through a lightweight LoRA pipeline. Experiments on three challenging real-world tasks show that Rollback improves success rates over the strongest baseline by an absolute 10.67%, 11.11%, and 11.85% on block extraction, ball tossing, and tabletop curling, respectively, while achieving more stable improvement throughout training. Although initialized from teleoperated demonstrations far below human level, Rollback surpasses the average success rate of human players on both ball tossing and tabletop curling within 8 hours of training.
