Latent Dynamics · 2026-10-07
Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models
Fernando Martinez, Tao Li, Yingdong Lu, Juntao Chen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
CASTLE is an offline-training, online-in-context guidance method for fully decentralized (independent) multi-agent reinforcement learning. Each agent uses two frozen world models: a Local Dynamics World Model (LDWM) for predictive local dynamics/partial observability and a Semantic-Social World Model (SSWM) trained on counterfactual simulator rollouts for action-specific short-horizon task/social consequences. Logits from these predictions guide an independent PPO policy without communication or a centralized critic.
Why it matters
In fully decentralized MARL, scalar rewards are ambiguous because poor return could come from the agent’s own action or from teammates/opponents’ responses. The authors propose prospectively comparing counterfactual consequences of candidate ego actions, using frozen, locally queried world models, to provide actionable guidance while keeping execution fully independent (no centralized critic or communication).
Method
- Introduce CASTLE: offline pretrain two frozen world models per agent—LDWM for local dynamics/partial observability and SSWM for action-conditioned short-horizon task and social consequences.
- Train SSWM from counterfactual simulator rollouts: restore logged rollout states, apply each candidate ego action, and roll out with teammates/opponents sampling from behavior policies to create targets.
- Online: run independent PPO where the actor logits are guided by LDWM context plus transient consequence guidance derived from frozen SSWM tokens; no world-model parameter updates online and no centralized critic or inter-agent communication.
Abstract (from arXiv)
Fully decentralized multi-agent reinforcement learning (MARL), also referred to as independent learning, requires each agent to learn and act using only its local information and experience, without a centralized critic or inter-agent communication. Such a stringent information structure renders the conventional reward signal ambiguous. A poor return may result from an ineffective ego action, an incompatible teammate response, or an effective opponent response, yet scalar rewards alone do not reveal which explanation is responsible. We argue that agents can learn more effectively by prospectively comparing the consequences of candidate actions rather than diagnosing failures only from realized returns. We introduce CASTLE (Counterfactual Action-conditioned Semantic Tokens for Local Execution in Decentralized MARL), an offline-training, online-in-context guidance framework with two complementary world models. A Local Dynamics World Model, offline pre-trained over agents' local trajectories, summarizes the agent's local trajectory dynamics and partial observability, while a Semantic-Social World Model predicts compact short-horizon task and social consequences for each candidate ego action. The latter is trained from counterfactual simulator rollouts that expose plausible teammate and opponent responses to alternative actions taken from the same logged rollout state. During online learning and execution, both world models remain frozen and are queried by agents using only locally available information. Their prediction logits provide in-context guidance to an independent PPO policy. Across 30 matched seeds on Tag, Spread, and Adversary in the benchmark multi-particle environments, our proposed CASTLE achieves the highest mean final score among the evaluated methods, exceeding the strongest baseline on each task by 10.67, 6.46, and 0.33 normalized points, respectively.
Related papers
- Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling
- MA-JEPA: Joint-Embedding World Models for Multi-Agent Reinforcement Learning
- LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces
- Player-1: Learning to Play from Synthetic Experience
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC
Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models
Preprint posted: 2026-10-06 · Original record ↗
- arxiv
- 2610.07704
Explicit source versions (1)
- v1 · 2026-10-06
- Preprint available
Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Fully decentralized multi-agent reinforcement learning (MARL), also referred to as independent learning, requires each agent to learn and act using only its local information and experience, without a centralized critic or inter-agent communication. Such a stringent information structure renders the conventional reward signal ambiguous. A poor return may result from an ineffective ego action, an incompatible teammate response, or an effective opponent response, yet scalar rewards alone do not reveal which explanation is responsible. We argue that agents can learn more effectively by prospectively comparing the consequences of candidate actions rather than diagnosing failures only from realized returns. We introduce CASTLE (Counterfactual Action-conditioned Semantic Tokens for Local Execution in Decentralized MARL), an offline-training, online-in-context guidance framework with two complementary world models. A Local Dynamics World Model, offline pre-trained over agents' local trajectories, summarizes the agent's local trajectory dynamics and partial observability, while a Semantic-Social World Model predicts compact short-horizon task and social consequences for each candidate ego action. The latter is trained from counterfactual simulator rollouts that expose plausible teammate and opponent responses to alternative actions taken from the same logged rollout state. During online learning and execution, both world models remain frozen and are queried by agents using only locally available information. Their prediction logits provide in-context guidance to an independent PPO policy. Across 30 matched seeds on Tag, Spread, and Adversary in the benchmark multi-particle environments, our proposed CASTLE achieves the highest mean final score among the evaluated methods, exceeding the strongest baseline on each task by 10.67, 6.46, and 0.33 normalized points, respectively.