← AI Techniques Radar
Inference efficiency

Coordinated Mixed-Precision KV Cache Quantization for Multi-Agent LLMs

Allocate different quantization bit-widths to simultaneously resident KV-cache blocks across multiple agents using an online sensitivity estimate and a shared controller that optimizes under a global memory budget.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

Multi-agent LLM systems can become memory-limited because several agents may hold KV caches at the same time; treating each agent independently with uniform quantization can harm collective reasoning. Coordinated allocation targets the total memory constraint while prioritizing precision where it matters most.

What changed

Instead of applying a single-model or per-agent quantization policy independently, the method (1) estimates per-layer sensitivity online from recent attention-related tensors without calibration, trial quantization, or extra model passes, and (2) performs joint bit-width allocation across agents under a shared byte-level capacity constraint using a centralized controller.

Where it helps

  • Multi-agent LLM systems with concurrent KV caches
  • Memory-constrained inference where KV cache dominates memory usage
  • Serving multiple agents whose precision needs differ across layers
  • Dynamic multi-agent runs where agents start/finish over time

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

Reported benefits are based on the described experimental setup and evaluated backbones/benchmarks; the approach assumes the existence of a way to track which KV-cache blocks are newly quantized versus already present, and it relies on the stated online sensitivity proxy.

Related World Models papers

Sources