← AI Techniques Radar
Retrieval & memory

Binary + int8 Embedding Quantization with Binary Fast Retrieval and float32 Rescoring

Quantize float32 sentence/document embeddings into (1) binary (1-bit-per-dimension) for very fast approximate search via Hamming distance, and optionally (2) scalar int8 embeddings (8-bit-per-dimension) for a compact candidate store. Then use a rescoring step: compare the float32 query embedding against the retrieved candidates (binary or int8) using dot-product to recover most of the original retrieval quality while keeping indices small.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

The approach targets embedding-storage and search scaling issues by reducing memory, disk usage, and retrieval-time computation. The source describes that binary search enables efficient distance computation, and rescoring with float32 improves performance retention versus using only the quantized retrieval signal.

What changed

Instead of searching directly with float32 embeddings, the system stores quantized representations (binary for the main index, int8 for compact on-disk candidate retrieval) and adds a post-retrieval rescoring stage using the original float32 query embedding.

Where it helps

  • Large-scale semantic search with memory/disk constraints
  • High-throughput retrieval where fast candidate generation is needed
  • Production systems that require lower latency while preserving retrieval quality via rescoring
  • Vector databases that support binary or scalar quantized indexing
  • Hybrid retrieval pipelines for document corpora

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

Performance depends on design choices and calibration for scalar quantization (e.g., the source notes calibration dataset/ranges affect int8 bucket definitions). Quality also relies on having an appropriate candidate set size for rescoring and a suitable rescore multiplier/top-k strategy.

Sources