Coordinated Mixed-Precision KV Cache Quantization for Multi-Agent LLMs
Allocate different quantization bit-widths to simultaneously resident KV-cache blocks across multiple agents using an online sensitivity estimate and a shared controller that optimizes under a global memory budget.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
Multi-agent LLM systems can become memory-limited because several agents may hold KV caches at the same time; treating each agent independently with uniform quantization can harm collective reasoning. Coordinated allocation targets the total memory constraint while prioritizing precision where it matters most.
What changed
Instead of applying a single-model or per-agent quantization policy independently, the method (1) estimates per-layer sensitivity online from recent attention-related tensors without calibration, trial quantization, or extra model passes, and (2) performs joint bit-width allocation across agents under a shared byte-level capacity constraint using a centralized controller.
Where it helps
- Multi-agent LLM systems with concurrent KV caches
- Memory-constrained inference where KV cache dominates memory usage
- Serving multiple agents whose precision needs differ across layers
- Dynamic multi-agent runs where agents start/finish over time
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
Reported benefits are based on the described experimental setup and evaluated backbones/benchmarks; the approach assumes the existence of a way to track which KV-cache blocks are newly quantized versus already present, and it relies on the stated online sensitivity proxy.
Related World Models papers
Sources
- Coordinated Mixed-Precision KV Cache Quantization for Efficient LLM-based Multi-agent Systems ↗
arXiv · primary · Checked 2026-11-10
