← AI Techniques Radar
Agents & tool use

Unified modular ASR + diarization + speculative decoding via a single custom inference handler

Build a single inference endpoint handler that chains an ASR model, an optional diarization pipeline over ASR outputs, and an optional speculative decoding path using an assistant (smaller) model whose decoder matches the main model’s decoder architecture.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

It enables adding diarization and assisted (speculative) generation to an ASR deployment while still exposing one API surface, with configurable model selection and inference parameters handled inside the endpoint.

What changed

Instead of deploying ASR alone, the approach introduces a custom inference handler that (1) loads an ASR model, (2) optionally loads a diarization model (Pyannote) for speaker partitioning, and (3) optionally loads an assistant causal language model to drive speculative decoding validated by the main model; all are controlled via endpoint configuration.

Where it helps

  • Speech transcription deployments that also need speaker diarization
  • Production ASR endpoints that require a single unified API
  • Latency-focused ASR for short utterances using assisted generation
  • Configurable pipelines where ASR-only or ASR+diarization are both needed

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

Speculative decoding has constraints: the assistant model must have a decoder architecture compatible with the main model, and the batch size must be 1.

Sources