Binary + int8 Embedding Quantization with Binary Fast Retrieval and float32 Rescoring
Quantize float32 sentence/document embeddings into (1) binary (1-bit-per-dimension) for very fast approximate search via Hamming distance, and optionally (2) scalar int8 embeddings (8-bit-per-dimension) for a compact candidate store. Then use a rescoring step: compare the float32 query embedding against the retrieved candidates (binary or int8) using dot-product to recover most of the original retrieval quality while keeping indices small.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
The approach targets embedding-storage and search scaling issues by reducing memory, disk usage, and retrieval-time computation. The source describes that binary search enables efficient distance computation, and rescoring with float32 improves performance retention versus using only the quantized retrieval signal.
What changed
Instead of searching directly with float32 embeddings, the system stores quantized representations (binary for the main index, int8 for compact on-disk candidate retrieval) and adds a post-retrieval rescoring stage using the original float32 query embedding.
Where it helps
- Large-scale semantic search with memory/disk constraints
- High-throughput retrieval where fast candidate generation is needed
- Production systems that require lower latency while preserving retrieval quality via rescoring
- Vector databases that support binary or scalar quantized indexing
- Hybrid retrieval pipelines for document corpora
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
Performance depends on design choices and calibration for scalar quantization (e.g., the source notes calibration dataset/ranges affect int8 bucket definitions). Quality also relies on having an appropriate candidate set size for rescoring and a suitable rescore multiplier/top-k strategy.
Sources
- Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval ↗
Hugging Face · primary · Published 2024-03-22 · Checked 2026-11-10
