Publication Radar · Source metadata

Optimal Value Inference for Reinforcement Learning

Nan Lu, Ethan Lee, Ruicheng Ao, David Simchi-Levi, Junwei Lu

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-09 · Indexed by Pub-AI: 2026-10-31 UTC

preprint

Optimal Value Inference for Reinforcement Learning

Nan Lu, Ethan Lee, Ruicheng Ao, David Simchi-Levi, Junwei Lu

Preprint posted: 2026-09-09 · Original record ↗

arxiv
2609.09981
Explicit source versions (2)
  • v3 · 2026-10-29
  • v4 · 2026-11-03
  1. Explicit source version detected · v3

    Source date: 2026-10-29 · First detected: 2026-10-31 UTC · Historical baseline

    Source evidence (1)
  2. Preprint available

    Source date: 2026-09-09 · First detected: 2026-10-31 UTC · Historical baseline

    Source evidence (2)
  3. Explicit source version detected · v4

    Source date: 2026-11-03 · First detected: 2026-11-13 UTC

    Source evidence (1)
Abstract · arxiv

We study offline inference for the optimal value in reinforcement learning under finite state and action spaces. Two new nuisances are derived as fixed points of a self-induced Bellman equation, in which we approximate the maximum Bellman operator by its softmax correspondence. We propose a debiased estimator through the Neyman orthogonality and establish its asymptotic normality under diverging horizons even when the behavior policy changes with time, as long as the nuisances have the statistical rates that can be achieved by many machine learning methods. We provide a concrete estimating procedure for these nuisances and show they can lead to valid inference. Synthetic experiments validate the numerical performance of our inference method, and we implement it in real-life decision-making problems, including bike repositioning and AI agentic tool use.