Latent Dynamics · 2026-11-03

Stand-In Effects and Pooling Amplification: Lessons from Biological World Models for Testing Action Effects

Bowen Xu, Boyu Chen, Jiaying Li

arXiv:2611.01666PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors show that common action-conditioning diagnostics for world models can report nonexistent “action effects” due to autocorrelated designs (future actions acting as stand-ins for history) and due to pooling amplification when many sites share one action sequence. They propose a design-based conditional randomization test using e-values (valid in finite samples under a known design law) and show tempering is a statistic choice that can fail, while fixed grids/mixtures can recover power.

Why it matters

Action-conditioned world models are often evaluated by checking prediction loss when actions are replaced/removed. The paper argues these diagnostics can be invalid in the sense of reporting false effects or losing power under pooling, and provides a validity-preserving alternative tied to the experiment’s design law. This matters for using world models trained on randomized experiments to answer interventional questions about action causal effects.

Method

  • Use a design-law-based conditional randomization test written as an e-value; validity is stated to hold in finite samples for known design laws when the resampler conditions on the full state the design used.
  • Identify two failures in learned action diagnostics: stand-in effects from autocorrelated designs and pooling amplification when many sites share one action sequence.
  • Study tempering/mixtures of tempered statistics; mixtures over a fixed grid are claimed to retain power more reliably than single estimated/tuned factors.

Limitation

The paper’s validity statements are conditional on reconstructing the design law (a “working law”) from published protocol and acquisition code; approximate laws are treated separately in §4.

Abstract (from arXiv)

Whether an action-conditioned world model responds to actions is usually checked by replacing or removing the actions and measuring the loss in accuracy. Taking biological experiments as the guiding case, we show that such diagnostics report action effects where there are none: under autocorrelated designs, future actions stand in for history the model does not see. When the design law is known, a conditional randomization test written as an e-value is valid in finite samples whatever the model. A second failure arises whenever many sites share one action sequence, as in world models: the pooled score turns small model errors along the action contrast into an e-value near zero, so a valid test loses its power although the effect is real. Tempering the score changes the bet, not the validity. The log e-value is concave in the tempering factor and zero at zero, so too small a factor costs little and too large a one can cost everything; every single factor we tried, estimated or tuned, failed in some setting, and mixtures over a fixed grid never did. Our own preregistered analysis of optogenetic ERK experiments failed by pooling; a tempered statistic, registered before the data were obtained, then rejected the null on a never-opened field (E = 1.1e10). In three registered tests on models trained by others, the pooled test rejected in 0 of 96 units of a diffusion model, and reached the power bar in 10 of 26 Atari games under IRIS's token likelihood and 1 of 6 control tasks under TD-MPC2; mixtures fixed in advance reached it in 96 units, 24 games and 6 tasks. On recorded robot video with iVideoGPT the mixture rejected in 155 of 159 trajectories; the pooled test did not collapse there (143). Under exact nulls the statistics made 14 rejections in 26,664 tests.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-01 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

Stand-In Effects and Pooling Amplification: Lessons from Biological World Models for Testing Action Effects

Bowen Xu, Boyu Chen, Jiaying Li

Preprint posted: 2026-11-01 · Original record ↗

arxiv
2611.01666
Explicit source versions (2)
  • v1 · Date not supplied
  • v4 · 2026-11-11
  1. Preprint available

    Source date: 2026-11-01 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (2)
  2. Explicit source version detected · v4

    Source date: 2026-11-11 · First detected: 2026-11-14 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Whether an action-conditioned world model responds to actions is usually checked by replacing or removing the actions and measuring the loss in accuracy. Taking biological experiments as the guiding case, we show that such diagnostics report action effects where there are none: under autocorrelated designs, future actions stand in for history the model does not see. When the design law is known, a conditional randomization test written as an e-value is valid in finite samples whatever the model. A second failure arises whenever many sites share one action sequence, as in world models: the pooled score turns small model errors along the action contrast into an e-value near zero, so a valid test loses its power although the effect is real. Tempering the score changes the bet, not the validity. The log e-value is concave in the tempering factor and zero at zero, so too small a factor costs little and too large a one can cost everything; every single factor we tried, estimated or tuned, failed in some setting, and mixtures over a fixed grid never did. Our own preregistered analysis of optogenetic ERK experiments failed by pooling; a tempered statistic, registered before the data were obtained, then rejected the null on a never-opened field (E = 1.1e10). In three registered tests on models trained by others, the pooled test rejected in 0 of 96 units of a diffusion model, and reached the power bar in 10 of 26 Atari games under IRIS's token likelihood and 1 of 6 control tasks under TD-MPC2; mixtures fixed in advance reached it in 96 units, 24 games and 6 tasks. On recorded robot video with iVideoGPT the mixture rejected in 155 of 159 trajectories; the pooled test did not collapse there (143). Under exact nulls the statistics made 14 rejections in 26,664 tests.