Stateful agent verification via terminal database state and side-effect executable checks (ThinkingBox-style)
Run an agent in an isolated, stateful sandbox where the evaluation judges the terminal backend state and extracted side effects (e.g., what records were changed) against an executable required end state, rather than trusting the agent’s final text or tool-call formatting. Repeat each task multiple times from a clean initial backend to measure consistency.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
The source claims that tool calls and seemingly correct narratives are only proxies: an agent can terminate cleanly and still leave wrong values, unintended extra effects, or miss required state changes. Verifying against database evidence is presented as the basis for detecting these gaps.
What changed
Evaluation is shifted from text/log-level signals (final response and whether tool calls are well-formed) to backend evidence: after isolated tool execution, a side-effect extractor derives what changed and deterministic judges compare that to required end states; tasks are also repeated from identical clean state to assess repeatability.
Where it helps
- Checking that an agent correctly updates ticket/order status in a database
- Verifying refunds/entitlements via required state transitions rather than narrative correctness
- Regression testing agent workflows before committing irreversible actions
- Measuring agent repeatability by rerunning tasks from identical initial state
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The approach depends on having an executable oracle for the required end state and the ability to observe/derive backend state and side effects from the agent’s tool interactions; it is also framed as a benchmark/sandbox method rather than a direct guarantee for all real deployments.
Sources
- The Agent Said It Was Done. The Database Disagreed. ↗
Hugging Face · primary · Published 2026-10-03 · Checked 2026-11-10
