Self-speculative decoding using early-exit layers with shared KV cache
Generate text by drafting tokens from early layers of the same LLM (via an early-exit point), then verify those drafted tokens using deeper layers. Intermediate-layer logits are made usable by unembedding through the LM head, enabled through a training recipe that teaches earlier layers to predict next tokens.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
It aims to reduce inference cost by reusing the model for both drafting and verification, avoiding a separate draft model, and leveraging cache reuse so verification can skip redundant computation.
What changed
Instead of using two separate models for speculative decoding (small draft model + large verifier), the method uses the LLM’s own early layers to draft and the same LLM’s later layers to verify, with early-exit halting and intermediate-layer unembedding support.
Where it helps
- Low-latency text generation in real-time systems
- Running large decoder-only LLMs with reduced hardware overhead
- Domain-adapted generation where the checkpoint was fine-tuned with the LayerSkip recipe
- Serving pipelines that can benefit from cache reuse across draft/verify stages
- Experiments comparing against standard (two-model) speculative decoding
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
Speed and correctness depend on having a checkpoint trained for this behavior; intermediate-layer logits generally cannot be unembedded meaningfully unless the model received the corresponding training modifications. The method also introduces a tuning trade-off between how early to exit and draft accuracy.
Sources
- Faster Text Generation with Self-Speculative Decoding ↗
Hugging Face · primary · Published 2024-11-20 · Checked 2026-11-10
