Driving · 2026-10-16

Constrained Multi-Agent Reinforcement Learning in Imagination

Raz Shmueli, Shahaf S. Shperberg

arXiv:2610.21319PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

CA-MAMBA is a constrained multi-agent RL method that extends a multi-agent world-model learner (MAMBA) with per-agent cost prediction. By training policies with reward and cost in imagination and setting the cost Lagrange multiplier from measured real episode cost, the authors report that model-free constrained baselines need at least an order of magnitude more interactions to match CA-MAMBA’s cost and task performance on most SMAC tasks.

Why it matters

It targets interaction efficiency for constrained multi-agent RL by moving most constraint learning into a learned simulator. The paper also analyzes when constraints learned in imagination transfer to the real system via an imagined-to-real cost decomposition, and tests controlled multiplier settings to assess whether model cost estimates or the learned price limit performance.

Method

  • Extend MAMBA with per-agent cost heads in an RSSM-style multi-agent world model; train reward and cost prediction in imagination.
  • Train per-agent policies on imagined reward and cost with a shared team Lagrange multiplier; update the multiplier using measured episode cost.
  • Use message passing (attention neighborhoods) so agents’ model interactions determine predicted team cost, and provide an imagined-to-real cost gap decomposition.

Limitation

The effect of communication on a trained model is treated as an open experiment in the main text (the appendix notes a neighborhood change also changes what deterministic states summarize, which its statement does not cover).

Abstract (from arXiv)

Predicting cost in a multi-agent world model moves most constrained policy updates into imagination: the stronger model-free baseline needs $10$ to $20$ times CA-MAMBA's interactions on the median SMAC configuration, and CA-MAMBA earns $2.8$ to $12$ times the baselines' MAMuJoCo reward at equal budget. In controlled tests on one task per benchmark, pricing the model's cost advantages at parity more than halves a cost-blind learner's cost and meets a tight continuous-control limit the learned multiplier misses, suggesting the model's cost estimates are not the bottleneck.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-15 · Indexed by Pub-AI: 2026-10-16 UTC

preprint

Constrained Multi-Agent Reinforcement Learning in Imagination

Raz Shmueli, Shahaf S. Shperberg

Preprint posted: 2026-10-15 · Original record ↗

arxiv
2610.21319
Explicit source versions (1)
  • v1 · 2026-10-15
  1. Preprint available

    Source date: 2026-10-15 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Predicting cost in a multi-agent world model moves most constrained policy updates into imagination: the stronger model-free baseline needs $10$ to $20$ times CA-MAMBA's interactions on the median SMAC configuration, and CA-MAMBA earns $2.8$ to $12$ times the baselines' MAMuJoCo reward at equal budget. In controlled tests on one task per benchmark, pricing the model's cost advantages at parity more than halves a cost-blind learner's cost and meets a tight continuous-control limit the learned multiplier misses, suggesting the model's cost estimates are not the bottleneck.