← AI Techniques Radar
Training & fine-tuning

Straight-Through Estimator (STE) for gradient flow through non-differentiable rounding in quantized LLM layers

Use a Straight-Through Estimator to enable backpropagation through quantization steps that include non-differentiable rounding. In BitLinear-style ternary weight quantization, the forward pass rounds weights to discrete values (e.g., clamped {-1, 0, 1}); during backpropagation, the STE approximates the gradient of the rounding operation as if it were an identity function so gradients can still update the underlying full-precision weights.

Discovered 2026-11-10 · Sources checked 2026-11-10

This summary was generated with AI assistance.

Why it matters

Extreme or low-bit quantization introduces discrete operations that are non-differentiable, which can otherwise block gradient-based optimization. STE provides a practical workaround so quantized/discretized layers can be trained using standard gradient methods.

What changed

Instead of treating rounding as a gradient-stopping operation, the method propagates gradients through the rounding step by approximating its derivative as 1 (identity), implemented via a straight-through approach (described in the source as using a detach-style trick).

Where it helps

  • Fine-tuning models with ternary (three-value) weight quantization
  • Training quantized linear layers where weight discretization uses round()
  • Adapting LLM layers to extreme weight/activation quantization while keeping backprop workable
  • Implementing STE-based gradient flow in custom quantization kernels or layers

Evidence & reproduction

Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.

Limitations & open questions

The source describes the approach as approximating gradients through rounding; this is an approximation and depends on the chosen quantization scheme (e.g., ternary weights and 8-bit activations) and the STE formulation.

Sources