Video World Models · 2026-11-04

Trusting Imagined Futures: Auditing Imagined Rewards for VLA Post-Training

Zicheng Kong, Zhenbo Xu, Haoyan Zhang, Jinghan Yang, Zhenzhen Li, Yiwei Ru, Liuyu Xiang, WanTing Xu, Min Ren, Zhaofeng He

arXiv:2611.04873PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Trusting Imagined Futures (TIF) is an offline supervision selection method for vision-language-action (VLA) post-training that audits reward predictions from a video world model using execution feedback. A multidimensional reward model predicts utility and five execution-quality dimensions for candidate ranking, while an auditor estimates overestimation risk. TIF replaces the current supervision only when predicted advantage exceeds a margin and audit risk is below a threshold.

Why it matters

VLA imitation post-training is sensitive to supervision quality: selecting candidates based on imagined reward can favor trajectories with poor execution. TIF aims to reduce this by gating reward-based replacements using an auditor trained from discrepancies between imagined reward predictions and matched real execution outcomes.

Method

  • Train a multidimensional reward model from matched execution outcomes and imagined videos; it predicts five execution-quality dimensions and a utility score for candidate ranking.
  • Train a risk auditor to identify overestimation patterns where predicted utility is high but execution reference utility is low, using predicted uncertainty and imagined-video diagnostics.
  • In selection, replace Source with the highest-predicted-utility candidate only if predicted advantage exceeds a margin and audit risk is <= a threshold; otherwise retain Source, then behavior-clone on retained real trajectories.
Abstract (from arXiv)

Video world models enable candidate actions to be evaluated through imagined outcomes, but high predicted rewards can favor trajectories with poor execution quality. We introduce Trusting Imagined Futures (TIF), an offline supervision selection method for vision-language-action (VLA) post-training that audits predicted rewards using execution feedback. A multidimensional reward model predicts a utility score for candidate ranking and five dimensions of execution quality. An auditor uses reward predictions and uncertainty to identify overestimation patterns learned from paired imagined videos and real executions. TIF accepts the candidate with the highest predicted utility when its advantage over the existing trajectory exceeds a margin and its audit risk passes a threshold; otherwise, it retains the existing supervision. The retained real trajectories then supervise behavior cloning. Across four LIBERO suites and three training seeds, TIF achieves a mean success rate of 76.25%, improving over training on the existing trajectories by 3.00 percentage points with the same trajectory count and training budget. Adding audit gating yields 1.47 percentage points with the reward model and advantage margin held fixed. The gains persist against advantage ranking and logistic regression risk selection at matched replacement counts. On recorded videos from three real robot tasks, TIF increases clean completion among selected trajectories from 58.33% to 70.00% across 60 candidate pools.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-04 · Indexed by Pub-AI: 2026-11-04 UTC

preprint

Trusting Imagined Futures: Auditing Imagined Rewards for VLA Post-Training

Zicheng Kong, Zhenbo Xu, Haoyan Zhang, Jinghan Yang, Zhenzhen Li, Yiwei Ru, Liuyu Xiang, WanTing Xu, Min Ren, Zhaofeng He

First announced by arXiv: 2026-11-04 · Original record ↗

arxiv
2611.04873
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-04 · First detected: 2026-11-04 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.04873v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Video world models enable candidate actions to be evaluated through imagined outcomes, but high predicted rewards can favor trajectories with poor execution quality. We introduce Trusting Imagined Futures (TIF), an offline supervision selection method for vision-language-action (VLA) post-training that audits predicted rewards using execution feedback. A multidimensional reward model predicts a utility score for candidate ranking and five dimensions of execution quality. An auditor uses reward predictions and uncertainty to identify overestimation patterns learned from paired imagined videos and real executions. TIF accepts the candidate with the highest predicted utility when its advantage over the existing trajectory exceeds a margin and its audit risk passes a threshold; otherwise, it retains the existing supervision. The retained real trajectories then supervise behavior cloning. Across four LIBERO suites and three training seeds, TIF achieves a mean success rate of 76.25%, improving over training on the existing trajectories by 3.00 percentage points with the same trajectory count and training budget. Adding audit gating yields 1.47 percentage points with the reward model and advantage margin held fixed. The gains persist against advantage ranking and logistic regression risk selection at matched replacement counts. On recorded videos from three real robot tasks, TIF increases clean completion among selected trajectories from 58.33% to 70.00% across 60 candidate pools.