Driving · 2026-11-03

Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

Weixiang Zhao, Zhihao Liu, Guancheng Li, Yanyan Zhao, Bing Qin, Ting Liu

arXiv:2611.02815PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors diagnose predictive safety guardrails as a prediction-to-action pipeline. Across seven policy models and two safety environments, stronger world-model prediction involvement is not monotonic for safety, can reduce task completion, and increases inference cost. Predictions are also insufficiently reliable, especially at longer horizons, and safety improvements can persist even with corrupted predictions—suggesting a prediction-to-action gap.

Why it matters

If predictive guardrails fail due to unreliable forecasts or because decision making does not faithfully use predicted risk, then evaluation on end-to-end safety alone can be misleading. The paper provides a structured diagnosis (integration, reliability, and decision faithfulness) and shows multiple failure modes that limit how much predictive foresight improves safety and task utility.

Method

  • They vary world-model prediction involvement via three integration settings: No Prediction, Prediction-Informed, and Prediction-Constrained, and evaluate safety/utility/cost.
  • They independently evaluate prediction reliability on recorded trajectories using multiple prediction models while varying horizon, policy information, and prediction mode.
  • They assess decision faithfulness by perturbing predicted content and using sandbox-verified predictions, testing whether outcomes depend on semantic validity vs mere presence.

Limitation

The RQ1 evaluations use specific backbone models (Qwen3.5-27B, GLM-5.3-Flash, Gemini-3.7-Flash, GPT-5.6-Terra) and two safety environments (HarnessAudit and IS-Bench), and the paper reports results within their experimental setup.

Abstract (from arXiv)

Large language model agents increasingly take consequential actions in digital and physical environments, where safety risks may emerge only after multiple interaction steps. This has motivated world-model-driven predictive safety guardrails, which anticipate future consequences before execution. However, existing evaluations largely focus on end-to-end safety outcomes, leaving unclear whether safety gains arise from reliable prediction and faithful use of predicted risks. In this work, we systematically diagnose the prediction-to-action pipeline along three dimensions: how world-model predictions are integrated into decision making, whether the predictions themselves are reliable, and whether their content is faithfully used. Across seven policy models and two safety environments, we find that stronger prediction involvement does not yield monotonic safety gains and can reduce task completion while substantially increasing inference cost. We further find that world-model predictions remain insufficiently reliable, especially over longer horizons, where safety prediction degrades substantially relative to near-term forecasting. Finally, we find that safety gains can persist even under corrupted predictions, while locally verified predictions still leave subsequent safety violations, showing that predictive content is not faithfully incorporated into the decision process. Together, these findings expose a fundamental prediction-to-action gap in current predictive guardrails: producing more or seemingly accurate foresight is insufficient unless that foresight is both reliable and faithfully translated into safer decisions.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

Weixiang Zhao, Zhihao Liu, Guancheng Li, Yanyan Zhao, Bing Qin, Ting Liu

First announced by arXiv: 2026-11-03 · Original record ↗

arxiv
2611.02815
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.02815v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Large language model agents increasingly take consequential actions in digital and physical environments, where safety risks may emerge only after multiple interaction steps. This has motivated world-model-driven predictive safety guardrails, which anticipate future consequences before execution. However, existing evaluations largely focus on end-to-end safety outcomes, leaving unclear whether safety gains arise from reliable prediction and faithful use of predicted risks. In this work, we systematically diagnose the prediction-to-action pipeline along three dimensions: how world-model predictions are integrated into decision making, whether the predictions themselves are reliable, and whether their content is faithfully used. Across seven policy models and two safety environments, we find that stronger prediction involvement does not yield monotonic safety gains and can reduce task completion while substantially increasing inference cost. We further find that world-model predictions remain insufficiently reliable, especially over longer horizons, where safety prediction degrades substantially relative to near-term forecasting. Finally, we find that safety gains can persist even under corrupted predictions, while locally verified predictions still leave subsequent safety violations, showing that predictive content is not faithfully incorporated into the decision process. Together, these findings expose a fundamental prediction-to-action gap in current predictive guardrails: producing more or seemingly accurate foresight is insufficient unless that foresight is both reliable and faithfully translated into safer decisions.