AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent–Mantissa Refinement
Automatically refines inference precision at runtime by selecting per-layer, per-step formats for weights, activations, and the KV cache. A lightweight policy chooses the precision using runtime signals such as next-token distribution entropy and the magnitude of each layer’s hidden-state updates. Precision formats are generated from a shared floating-point base via multi-scale exponent/mantissa refinement, enabling multiple reduced-precision variants from a single underlying format.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
Fixed-precision quantization selected before inference can be suboptimal because precision needs vary across prompts, layers, and phases (e.g., prefill vs. decode). Runtime selection aims to better match precision to the local difficulty of generation signals and layer behavior while keeping inference efficient.
What changed
Instead of determining precision once during post-training/quantization, the method adapts precision continuously during inference for each layer and for each prefill/decode step, and it uses a multi-scale floating-point scheme to derive multiple reduced-precision formats from one base representation.
Where it helps
- LLM inference where prompt difficulty varies significantly
- Serving large models across diverse latency/quality tradeoffs
- Systems that support multi-format floating-point execution for quantized inference
- Deployments needing KV-cache precision adaptation during decode
- Inference pipelines that distinguish prefill vs. decode behavior
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The approach depends on having a runtime policy and a hardware/software path that supports the required multi-scale floating-point formats for weights/activations/KV cache; exact compatibility and overhead details are not provided in the supplied text.
Related World Models papers
Sources
- AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement ↗
arXiv · primary · Checked 2026-11-10
