Latent Dynamics · 2026-11-13
FOE-Dreamer: Deployment-Efficient Learning of Cyber Defense Policies in Operational Networks
Yinuo Du, Christopher Kiekintveld
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
FOE-Dreamer is a model-based RL framework for cyber defense that trains online and directly in an emulated operational network on cloud infrastructure, without a simulator stage. It uses a factored world model separating defender-driven, exogenous user activity, and opponent dynamics inferred from defender-observable signals. On an 8-host OpenStack testbed with GHOSTS user emulation and two scripted attacker profiles, it trains in 3 days on a single GPU and reduces episode loss by about 2× vs Rainbow and IQN under matched compute.
Why it matters
The authors target the simulator-fidelity gap and the sample-efficiency constraint when training directly in operational networks. By separating controllable vs exogenous vs opponent-driven latent variation and conditioning planning on an inferred opponent embedding, FOE-Dreamer aims to reduce variance during imagined-rollout planning while reserving real-network interaction for data collection. The evaluation uses an OpenStack-based, real-software, GHOSTS-style user-emulation testbed.
Method
- Trains a defender online and directly in an emulated operational network (OpenStack testbed) without a simulator stage; policy optimization uses imagined latent rollouts from a learned world model.
- Uses a factored world model that separates three latent-state variation sources: defender-driven (controllable), exogenous (benign user activity), and opponent (attacker) embedding inferred from defender-observable signals alone.
- Adopts an opponent model that infers attacker type from defender-observable signals and conditions imagined rollouts on the inferred type.
Limitation
Deferred analyses. A hyperparameter sensitivity sweep and a quantitative prevalence count of the failure modes above are deferred to follow-on work; both require additional training and evaluation cycles beyond the current compute budget.
Abstract (from arXiv)
Autonomous cyber defense is widely studied with deep reinforcement learning, but most work trains policies inside simulators because learning requires many trial-and-error interactions. Simulator-based training carries a fidelity gap: trained policies face authentication delays, partial action failures, service dependencies, and user traffic patterns that the simulator did not represent, and degrade on transfer. Training directly in the network being defended removes the gap but makes every interaction slow and costly, so sample efficiency becomes the binding constraint. Model-based reinforcement learning has reduced sample requirements on games and physical robots by orders of magnitude by planning inside a learned world model, but it has not been applied to defenders trained in operational networks. We present FOE-Dreamer (Factored Opponent-Enhanced Dreamer), a model-based reinforcement learning framework that trains a defender online and directly in a network on cloud infrastructure, without a simulator stage. FOE-Dreamer separates three sources of variation in the latent state: a controllable component driven by defender actions, an exogenous component driven by benign user activity, and an opponent embedding inferred from defender-observable signals alone. Separating the components reduces the variance in policy gradients during planning in imagination. Against two scripted attacker profiles with user activity emulated by the GHOSTS framework on an 8-host OpenStack testbed, FOE-Dreamer trains within a 3-day budget on a single GPU and achieves about 2$\times$ better episode loss than Rainbow and IQN baselines under matched compute. Both the factoring and the opponent model contribute to the gain, and the learned latents are consistent with the intended separation.
Related papers
- CyberWorld: World Models for Sample-Efficient Autonomous Cyber Defense
- DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents
- Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition
- Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models
- Deploying Physically and Epistemically Compliant Agents by Distilling Belief-Space Task and Motion Planners
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC
FOE-Dreamer: Deployment-Efficient Learning of Cyber Defense Policies in Operational Networks
First announced by arXiv: 2026-11-13 · Original record ↗
- arxiv
- 2611.14880
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.14880v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Autonomous cyber defense is widely studied with deep reinforcement learning, but most work trains policies inside simulators because learning requires many trial-and-error interactions. Simulator-based training carries a fidelity gap: trained policies face authentication delays, partial action failures, service dependencies, and user traffic patterns that the simulator did not represent, and degrade on transfer. Training directly in the network being defended removes the gap but makes every interaction slow and costly, so sample efficiency becomes the binding constraint. Model-based reinforcement learning has reduced sample requirements on games and physical robots by orders of magnitude by planning inside a learned world model, but it has not been applied to defenders trained in operational networks. We present FOE-Dreamer (Factored Opponent-Enhanced Dreamer), a model-based reinforcement learning framework that trains a defender online and directly in a network on cloud infrastructure, without a simulator stage. FOE-Dreamer separates three sources of variation in the latent state: a controllable component driven by defender actions, an exogenous component driven by benign user activity, and an opponent embedding inferred from defender-observable signals alone. Separating the components reduces the variance in policy gradients during planning in imagination. Against two scripted attacker profiles with user activity emulated by the GHOSTS framework on an 8-host OpenStack testbed, FOE-Dreamer trains within a 3-day budget on a single GPU and achieves about 2$\times$ better episode loss than Rainbow and IQN baselines under matched compute. Both the factoring and the opponent model contribute to the gain, and the learned latents are consistent with the intended separation.
