Straight-Through Estimator (STE) for gradient flow through non-differentiable rounding in quantized LLM layers
Use a Straight-Through Estimator to enable backpropagation through quantization steps that include non-differentiable rounding. In BitLinear-style ternary weight quantization, the forward pass rounds weights to discrete values (e.g., clamped {-1, 0, 1}); during backpropagation, the STE approximates the gradient of the rounding operation as if it were an identity function so gradients can still update the underlying full-precision weights.
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
Extreme or low-bit quantization introduces discrete operations that are non-differentiable, which can otherwise block gradient-based optimization. STE provides a practical workaround so quantized/discretized layers can be trained using standard gradient methods.
What changed
Instead of treating rounding as a gradient-stopping operation, the method propagates gradients through the rounding step by approximating its derivative as 1 (identity), implemented via a straight-through approach (described in the source as using a detach-style trick).
Where it helps
- Fine-tuning models with ternary (three-value) weight quantization
- Training quantized linear layers where weight discretization uses round()
- Adapting LLM layers to extreme weight/activation quantization while keeping backprop workable
- Implementing STE-based gradient flow in custom quantization kernels or layers
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The source describes the approach as approximating gradients through rounding; this is an approximation and depends on the chosen quantization scheme (e.g., ternary weights and 8-bit activations) and the STE formulation.
Sources
- Fine-tuning LLMs to 1.58bit: extreme quantization made easy ↗
Hugging Face · primary · Published 2024-09-18 · Checked 2026-11-10
