Latent Dynamics · 2026-10-12

Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

Oriol Corcoll, Qiyao Wei, Ciar\'{a}n Mark Gilligan-Lee

arXiv:2610.13504PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors introduce Explanatory World Models (EWM), an attribution framework that uses explanations of an observed outcome to redistribute credit to earlier events. They instantiate EWM with a frontier language model and a conservative hierarchical redistribution operator that recursively reallocates reward while preserving undiscounted episode return. Across Crafter, WordCraft, KuaiRand (offline/online), and an offline industry recommender evaluation, EWM improves credit assignment especially when rewards are highly delayed.

Why it matters

Standard RL assigns credit by temporal proximity, which can make delayed but important actions hard to learn from without many trajectories. EWM replaces this temporal prior with trajectory-dependent credit assignment derived from explanations, aiming to provide structured, reusable credit for learning from delayed outcomes. The paper evaluates both controlled long-horizon RL benchmarks and recommendation settings (offline and online), showing sizable gains when rewards are sparse/delayed and better offline Recall and behavioral proxies in an industry-scale recommender evaluation.

Method

  • Train an Explanatory World Model (EWM) with a frontier language model to generate an explanation graph connecting a phenomenon (outcome) to preceding events with attribution weights.
  • Convert explanations to learning signals using a conservative hierarchical reward redistribution operator (CHO) that recursively redistributes reward to explanation predecessors, terminating by max depth, no eligible predecessors, or low carried reward; it is conservative in total reward mass.
  • Integrate the redistribution as an attribution layer between observed rewards and RL updates: compute returns/advantages from redistributed rewards rather than the original reward stream; cache explanation graphs for offline learning.
Abstract (from arXiv)

Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-12 UTC

preprint

Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

Oriol Corcoll, Qiyao Wei, Ciarán Mark Gilligan-Lee

Preprint posted: 2026-10-09 · Original record ↗

arxiv
2610.13504
Explicit source versions (1)
  • v1 · 2026-10-09
  1. Preprint available

    Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.