← AI Techniques Radar
Inference efficiency

Self-speculative decoding using early-exit layers with shared KV cache

Generate text by drafting tokens from early layers of the same LLM (via an early-exit point), then verify those drafted tokens using deeper layers. Intermediate-layer logits are made usable by unembedding through the LM head, enabled through a training recipe that teaches earlier layers to predict next tokens.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

It aims to reduce inference cost by reusing the model for both drafting and verification, avoiding a separate draft model, and leveraging cache reuse so verification can skip redundant computation.

What changed

Instead of using two separate models for speculative decoding (small draft model + large verifier), the method uses the LLM’s own early layers to draft and the same LLM’s later layers to verify, with early-exit halting and intermediate-layer unembedding support.

Where it helps

  • Low-latency text generation in real-time systems
  • Running large decoder-only LLMs with reduced hardware overhead
  • Domain-adapted generation where the checkpoint was fine-tuned with the LayerSkip recipe
  • Serving pipelines that can benefit from cache reuse across draft/verify stages
  • Experiments comparing against standard (two-model) speculative decoding

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

Speed and correctness depend on having a checkpoint trained for this behavior; intermediate-layer logits generally cannot be unembedded meaningfully unless the model received the corresponding training modifications. The method also introduces a tuning trade-off between how early to exit and draft accuracy.

Sources