Memory-constrained RLHF fine-tuning using 8-bit base weights plus trainable low-rank adapters and adapter-disabling for reference logits
Fine-tune an LLM for RLHF by keeping the base model in 8-bit and training only low-rank adapter weights (via PEFT) while computing both active and reference model logits using the same model by temporarily disabling adapters (so two full copies are not required).
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
RLHF commonly needs both an “active” policy model and a “reference” model during optimization; maintaining two full model copies is memory-expensive. Using an 8-bit frozen base plus small trainable adapters reduces memory usage and enables the reference logits to be obtained through adapter enable/disable rather than duplicating full model parameters per device.
What changed
Instead of storing and updating full-precision weights (and maintaining separate reference and active model copies), the approach changes to: (1) load the base model in 8-bit, (2) add trainable low-rank adapters, and (3) compute reference and active logits by disabling adapters on the same underlying model.
Where it helps
- RLHF alignment for text generation where GPU memory is limited
- Optimizing a model toward subjective rewards (e.g., positive sentiment)
- Reducing the cost of RLHF fine-tuning on consumer GPUs
- Setting up PPO-style RLHF workflows with parameter-efficient fine-tuning
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The approach is described as having a training-speed downside (adapter-based fine-tuning adds additional computation). It also relies on an RLHF setup that can compute reference logits every optimization step using adapter toggling rather than full model duplication, which may impose constraints on the training implementation.
Sources
- Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU ↗
Hugging Face · primary · Published 2023-03-09 · Checked 2026-11-10
