← AI Techniques Radar
Inference efficiency

Assisted Generation (Speculative Decoding) with Quantized Draft and Target Models

Use a small “draft” model to propose multiple candidate tokens, then have the larger “target” model verify all candidates in parallel via a single forward pass. In the reported approach, both the draft and target are quantized (with choices compared between 8-bit SmoothQuant and 4-bit weight-only quantization) to improve the balance between memory bandwidth and compute overhead on Intel Xeon.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

Autoregressive decoding can become limited by memory bandwidth when tokens are generated one-by-one and the model is repeatedly reloaded from off-chip memory. Parallel verification of K candidate tokens reduces sequential autoregressive overhead, and quantization can further shift the bottleneck and reduce model loading/memory pressure.

What changed

Instead of standard token-by-token decoding with a single model, the method adds (1) an assistant/draft model that greedily generates K candidate tokens and (2) a speculative-style check by the target model that processes the K tokens together in parallel. The method also applies quantization to both draft and target models, comparing 8-bit SmoothQuant vs 4-bit weight-only quantization to choose configurations that best match the new bottleneck behavior.

Where it helps

  • Faster decoding for code generation LLMs
  • Low-latency text generation on CPU (Intel Xeon)
  • Speculative decoding using a smaller assistant model
  • Quantization-aware decoding pipelines

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

The reported gains depend on matching quantization choices to the stage/bottleneck: 4-bit weight-only quantization introduces dequantization compute overhead, which can reduce speedups when compute becomes the bottleneck (e.g., during parallel target verification).

Sources