Driving · 2026-11-02
When Does Test-Time Search Help World Action Models? An Empirical Audit of Candidate Selection and Simulator Integrity
Jun Liu, Dajie Sun, Zhenglun Kong, Zhenjiang Fan, Peiyan Dong, Fan Yang, Wenbin Zhangg, Geng Yuan, Xue Lin, Yanzhi Wang, Dong Huang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors audit test-time scaling for world-action models on Cosmos Policy with RoboCasa using a premise-first, pre-registered methodology. On clean data, a learned value selector is 1.66 pp worse than random; a hindsight oracle gains only 1.40 pp with an episode-level interval spanning zero. Token pruning yields at most 1.152× end-to-end speedup on the deployed path (1.253× decode-excluded), short of 1.5×, and they identify/fix a compute-dependent rendering corruption.
Why it matters
The paper argues that success-rate gains from test-time search/planning are only credible if (i) candidates have exploitable decision signal, (ii) nominal FLOPs reductions translate to real end-to-end speedup, and (iii) evaluation stays reliable under added compute. The authors’ audits show weak selection headroom, limited realized efficiency from token pruning, and a rendering corruption that depends on the inference path, motivating chained premise audits rather than single outcome metrics.
Method
- Pre-registered, premise-first audit of test-time scaling using (1) candidate selection premise checks, (2) statistical power gating for resolvable outcomes, and (3) render-integrity verification with a framebuffer-sentinel detector.
- For selection headroom, the authors replay all N=8 candidate actions per query state under a fixed continuation policy and estimate an outcome-level (non-deployable) hindsight oracle.
- For efficiency, they directly time the complete end-to-end action-inference query under whole-frame token pruning (deployed and decode-excluded paths) rather than composing component timings.
Limitation
Actions cover only the executed first 16 of the 32 generated steps. With two repeats per candidate, they cannot adjudicate per state whether stable separation is real or coincidental; graded (0.5-valued) states are indistinguishable from noise at this sample size. They also note the diagnostics cover four tasks and three episodes and do not speak to other candidate counts, sampling temperatures, or continuation policies.
Abstract (from arXiv)
World-action models increasingly apply test-time scaling, spending extra inference computation on candidate selection and adaptive allocation to improve control. Such gains are credible only when candidates carry exploitable signal, nominal compute savings become real speedup, and evaluation stays reliable. We audit all three on Cosmos Policy and RoboCasa with a premise-first, pre-registered methodology. On clean data, a learned value selector is 1.66 pp worse than random; a hindsight oracle gains only 1.40 pp, with an episode-level interval spanning zero; and 82% of states show identical candidate outcomes, leaving little selection headroom. Token pruning cuts per-forward FLOPs by up to 47% but yields at most a measured 1.15$\times$ end-to-end speedup on the deployed path (1.25$\times$ on a decode-excluded path), short of a 1.5$\times$ target, and our adaptive-allocation experiment is inconclusive under a power gate. We further identify and fix a silent rendering corruption whose rate rises sharply along compute-heavy planning paths while leaving the official direct-policy result unaffected, and release a detector, fix, and scripts. These findings show why test-time scaling should be audited as a chain of premises (decision signal, realized efficiency, evaluationintegrity) rather than by success rate alone. Conclusions are scoped to the tested configuration.
Related papers
- How Much Can Best-of-N Help a World Action Model? Measuring Selectable Headroom and Realizing It in Closed Loop
- When Low Prediction Error Misleads Planning: Diagnosing Representation, Dynamics, and Decision Failures in Latent World Models
- Agentic World Modeling for Agent Test-time Scaling without Environment Replica
- ActTime: Action-Conditioned World Modeling for Time Series
- One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-30 · Indexed by Pub-AI: 2026-11-02 UTC
When Does Test-Time Search Help World Action Models? An Empirical Audit of Candidate Selection and Simulator Integrity
Preprint posted: 2026-10-30 · Original record ↗
- arxiv
- 2610.41970
Explicit source versions (1)
- v1 · 2026-10-30
- Preprint available
Source date: 2026-10-30 · First detected: 2026-11-02 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.41970v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World-action models increasingly apply test-time scaling, spending extra inference computation on candidate selection and adaptive allocation to improve control. Such gains are credible only when candidates carry exploitable signal, nominal compute savings become real speedup, and evaluation stays reliable. We audit all three on Cosmos Policy and RoboCasa with a premise-first, pre-registered methodology. On clean data, a learned value selector is 1.66 pp worse than random; a hindsight oracle gains only 1.40 pp, with an episode-level interval spanning zero; and 82% of states show identical candidate outcomes, leaving little selection headroom. Token pruning cuts per-forward FLOPs by up to 47% but yields at most a measured 1.15$\times$ end-to-end speedup on the deployed path (1.25$\times$ on a decode-excluded path), short of a 1.5$\times$ target, and our adaptive-allocation experiment is inconclusive under a power gate. We further identify and fix a silent rendering corruption whose rate rises sharply along compute-heavy planning paths while leaving the official direct-policy result unaffected, and release a detector, fix, and scripts. These findings show why test-time scaling should be audited as a chain of premises (decision signal, realized efficiency, evaluationintegrity) rather than by success rate alone. Conclusions are scoped to the tested configuration.