Driving · 2026-10-16
Constrained Multi-Agent Reinforcement Learning in Imagination
Raz Shmueli, Shahaf S. Shperberg
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
CA-MAMBA is a constrained multi-agent RL method that extends a multi-agent world-model learner (MAMBA) with per-agent cost prediction. By training policies with reward and cost in imagination and setting the cost Lagrange multiplier from measured real episode cost, the authors report that model-free constrained baselines need at least an order of magnitude more interactions to match CA-MAMBA’s cost and task performance on most SMAC tasks.
Why it matters
It targets interaction efficiency for constrained multi-agent RL by moving most constraint learning into a learned simulator. The paper also analyzes when constraints learned in imagination transfer to the real system via an imagined-to-real cost decomposition, and tests controlled multiplier settings to assess whether model cost estimates or the learned price limit performance.
Method
- Extend MAMBA with per-agent cost heads in an RSSM-style multi-agent world model; train reward and cost prediction in imagination.
- Train per-agent policies on imagined reward and cost with a shared team Lagrange multiplier; update the multiplier using measured episode cost.
- Use message passing (attention neighborhoods) so agents’ model interactions determine predicted team cost, and provide an imagined-to-real cost gap decomposition.
Limitation
The effect of communication on a trained model is treated as an open experiment in the main text (the appendix notes a neighborhood change also changes what deterministic states summarize, which its statement does not cover).
Abstract (from arXiv)
Predicting cost in a multi-agent world model moves most constrained policy updates into imagination: the stronger model-free baseline needs $10$ to $20$ times CA-MAMBA's interactions on the median SMAC configuration, and CA-MAMBA earns $2.8$ to $12$ times the baselines' MAMuJoCo reward at equal budget. In controlled tests on one task per benchmark, pricing the model's cost advantages at parity more than halves a cost-blind learner's cost and meets a tight continuous-control limit the learned multiplier misses, suggesting the model's cost estimates are not the bottleneck.
Related papers
- Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models
- Latent Imagination for Parameterized Action Spaces
- World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
- Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents
- Train An Imaginative Agent
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-15 · Indexed by Pub-AI: 2026-10-16 UTC
Constrained Multi-Agent Reinforcement Learning in Imagination
Preprint posted: 2026-10-15 · Original record ↗
- arxiv
- 2610.21319
Explicit source versions (1)
- v1 · 2026-10-15
- Preprint available
Source date: 2026-10-15 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Predicting cost in a multi-agent world model moves most constrained policy updates into imagination: the stronger model-free baseline needs $10$ to $20$ times CA-MAMBA's interactions on the median SMAC configuration, and CA-MAMBA earns $2.8$ to $12$ times the baselines' MAMuJoCo reward at equal budget. In controlled tests on one task per benchmark, pricing the model's cost advantages at parity more than halves a cost-blind learner's cost and meets a tight continuous-control limit the learned multiplier misses, suggesting the model's cost estimates are not the bottleneck.
