Driving · 2026-10-06
Repeated-Measure Leakage, Distribution Shift, and Reliability under Partial Observation in Patient World Models
Arjun Subramanian
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors test leakage-resistant patient generalization and reliability for short-horizon gait digital-biomarker forecasting on GaitPDB. They build an evaluation ladder that progressively removes raw temporal overlap, same-recording familiarity, and same-patient familiarity before unseen-patient testing. GRU NMSE increases from 0.1227 (random-window) to 0.1393 (no raw overlap) to 0.1961 (patient holdout), and missingness degrades performance while MC-dropout variance decreases.
Why it matters
Before claims about longitudinal or intervention-aware patient world models, the authors argue that predictive states must generalize across patients, survive distribution shift and missing observations, and provide reliability signals that track failure. Their results show evaluation optimism from repeated-measure familiarity, degradations under internal shift and missingness, and a reliability mismatch where MC-dropout uncertainty can decrease when NMSE worsens.
Method
- Build a four-level repeated-measure evaluation ladder (random-window, blocked/no-overlap with zero-overlap checks, same-patient/new-recording, participant-held-out). Use hierarchical participant-level aggregation for participant-balanced evaluation.
- Stress-test generalization with leave-one-study-out (LOSO), a longer prediction gap (0.64s vs 0.16s), and partial observation corruptions (temporal masking, channel removal, additive Gaussian noise).
- Assess reliability via MC-dropout predictive variance and selective-prediction diagnostics; abstention thresholds chosen on training-side validation only.
Limitation
GaitPDB is a single public resource; its three source studies are not independent external cohorts and differ in protocol, demographics, and repeated-recording structure. The specific effect sizes should not be assumed to transport to other diseases, modalities, institutions, or datasets. The task predicts sub-second VGRF segments within a session, not longitudinal disease progression, treatment response, interventions, or causal effects.
Abstract (from arXiv)
Patient world models are increasingly proposed for longitudinal prediction, intervention-aware reasoning, and clinical-trial simulation. Causal or clinical intervention validity is distinct from predictive generalization and reliability; before making stronger claims, the underlying predictive state should generalize across patients, survive realistic shifts and missing observations, and expose failure through meaningful reliability signals. We evaluate these prerequisites in a deliberately narrow setting: short-horizon digital-biomarker forecasting from PhysioNet GaitPDB, comprising 165 participants, 306 recordings, and 51,129 context-future pairs. Using persistence, ridge, MLP, GRU, Transformer, and a compact JEPA-style predictor, we build an evaluation ladder that progressively removes raw temporal overlap, same-recording familiarity, and same-patient familiarity before testing unseen-patient generalization. For GRU, NMSE rises from 0.1227 under random-window splitting to 0.1393 after eliminating raw train-test overlap and to 0.1961 under patient holdout. Among 54 participants with repeated recordings, exposure to a different recording from the same patient improves GRU NMSE from 0.2177 to 0.1556, while a recording-excluded identity hypothesis is not supported at the participant level. Under participant-held-out evaluation, MLP and Transformer are statistically indistinguishable. Study shift, a four-times-longer prediction gap, and partial observation further degrade performance; under 50% temporal masking, Transformer NMSE rises to 0.611 while MC-dropout predictive variance falls. We do not claim a longitudinal or intervention-aware simulator. Instead, the results support a prerequisite evaluation stack of patient separation, repeated-measure controls, shift, missingness, and uncertainty validation before stronger patient-world-model claims are trusted.
Related papers
- PatientWM: Towards Generalizable Clinical World Models via Unified Patient States
- Same Predictions, Different Harms: Causal Auditing of Patient World Models
- Patient, Place, Prior (P$^3$): What Counts as Personalization in Medical World Models?
- Forecasting Patient Trajectories from the Electronic Health Records with Action-Conditioned World Models
- Understanding Trajectory Heterogeneity in Federated World Model Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC
Repeated-Measure Leakage, Distribution Shift, and Reliability under Partial Observation in Patient World Models
Preprint posted: 2026-10-03 · Original record ↗
- arxiv
- 2610.04778
Explicit source versions (1)
- v1 · 2026-10-03
- Preprint available
Source date: 2026-10-03 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Patient world models are increasingly proposed for longitudinal prediction, intervention-aware reasoning, and clinical-trial simulation. Causal or clinical intervention validity is distinct from predictive generalization and reliability; before making stronger claims, the underlying predictive state should generalize across patients, survive realistic shifts and missing observations, and expose failure through meaningful reliability signals. We evaluate these prerequisites in a deliberately narrow setting: short-horizon digital-biomarker forecasting from PhysioNet GaitPDB, comprising 165 participants, 306 recordings, and 51,129 context-future pairs. Using persistence, ridge, MLP, GRU, Transformer, and a compact JEPA-style predictor, we build an evaluation ladder that progressively removes raw temporal overlap, same-recording familiarity, and same-patient familiarity before testing unseen-patient generalization. For GRU, NMSE rises from 0.1227 under random-window splitting to 0.1393 after eliminating raw train-test overlap and to 0.1961 under patient holdout. Among 54 participants with repeated recordings, exposure to a different recording from the same patient improves GRU NMSE from 0.2177 to 0.1556, while a recording-excluded identity hypothesis is not supported at the participant level. Under participant-held-out evaluation, MLP and Transformer are statistically indistinguishable. Study shift, a four-times-longer prediction gap, and partial observation further degrade performance; under 50% temporal masking, Transformer NMSE rises to 0.611 while MC-dropout predictive variance falls. We do not claim a longitudinal or intervention-aware simulator. Instead, the results support a prerequisite evaluation stack of patient separation, repeated-measure controls, shift, missingness, and uncertainty validation before stronger patient-world-model claims are trusted.