Pub-AI.AI in Science
Search Pub-AIStart Here

Latent Dynamics · 2026-11-02

One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses

Junyu Huo, Zihao Wan, Ziqi Mao, Gouri Ginde

arXiv:2610.41130PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors report TetraState, a regression-testing protocol for learned video world models that tests paired possible/impossible items under four fixed model-native configurations. It replaces accuracy-only or single-verdict oracles with configuration-level outcome classes and outcome-transition comparisons across revisions, versioning the harness as part of the SUT. They show harness revisions can change many per-test labels while aggregate modal accuracy barely moves.

Why it matters

Accuracy-only or single-verdict checks can miss regressions from configuration-dependent behavior and harness faults that produce wrong verdicts without crashes. By using configuration-level outcome classes and per-test outcome transitions, TetraState aims to make regression testing sensitive to changes in correctness profiles across settings, and to treat the test harness (sampling, scoring, loss aggregation) as part of what gets versioned and tested.

Method

  • TetraState executes each paired possible/impossible test under four fixed model-native configurations and uses an outcome oracle with four classes (CP, CDP, CDF, CF) plus separate reporting for inconclusive (2–2) and incomplete executions.
  • For a revision r→r′, it compares per-test outcome transitions (summarizing configuration-dependent correctness patterns) and reports verdict-rate changes and the direction of per-test class changes.
  • The protocol versions the harness as part of the SUT identifier; executions record provenance and include build manifests and validation checks to prevent mixed SUT identifiers.

Limitation

Both revisions are post hoc case studies on one SUT, and the regression view was defined after observing them; they illustrate, but do not establish, detection power. Configuration dependence mixes model behavior with evidence availability, and semantic preservation remains an assumption to examine.

Abstract (from arXiv)

Learned world models ship as software components whose model versions, preprocessing, and test harnesses change between releases. Work on model updates already counts per-sample regressions, or negative flips, that an accuracy check hides. Two problems remain: a test can pass under one configuration and fail under another, and a harness fault can produce erroneous scores and wrong verdicts without a crash. We present TetraState, a regression-testing protocol for paired possible/impossible video tests. Each test runs under four fixed model-native configurations; its number of correct verdicts assigns one of four outcome classes. Inconclusive tests and failed executions are reported separately alongside the verdict rate. Revisions are compared by per-test outcome transitions, and the harness is versioned as part of the system under test. On all 506 IntPhys 2 Main tests and two heterogeneous systems, Cosmos 3 Nano and V-JEPA 2, two V-JEPA 2 harness revisions move modal accuracy by only 0.2 and 2.6 points yet change 11 and 214 test labels; in the second, accuracy rises although fewer tests are judged correctly, and 43 regressions keep the modal verdict, so a modal-verdict flip check misses them. These results outline a research agenda on configuration-level regression oracles for learned components.

Related papers