Unified modular ASR + diarization + speculative decoding via a single custom inference handler
Build a single inference endpoint handler that chains an ASR model, an optional diarization pipeline over ASR outputs, and an optional speculative decoding path using an assistant (smaller) model whose decoder matches the main model’s decoder architecture.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
It enables adding diarization and assisted (speculative) generation to an ASR deployment while still exposing one API surface, with configurable model selection and inference parameters handled inside the endpoint.
What changed
Instead of deploying ASR alone, the approach introduces a custom inference handler that (1) loads an ASR model, (2) optionally loads a diarization model (Pyannote) for speaker partitioning, and (3) optionally loads an assistant causal language model to drive speculative decoding validated by the main model; all are controlled via endpoint configuration.
Where it helps
- Speech transcription deployments that also need speaker diarization
- Production ASR endpoints that require a single unified API
- Latency-focused ASR for short utterances using assisted generation
- Configurable pipelines where ASR-only or ASR+diarization are both needed
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
Speculative decoding has constraints: the assistant model must have a decoder architecture compatible with the main model, and the batch size must be 1.
Sources
- Powerful ASR + diarization + speculative decoding with Hugging Face Inference Endpoints ↗
Hugging Face · primary · Published 2024-05-01 · Checked 2026-11-10
