Parameter-efficient fine-tuning: adapt billion-parameter models on a single GPU by training less than 1% of all weights.
Full fine-tuning updates every parameter in the model. For a 7B-parameter model, that means storing optimizer states, gradients, and a full copy of every weight — often requiring 60–120 GB of VRAM.
Parameter-Efficient Fine-Tuning (PEFT) methods freeze the original weights and train only a small number of additional parameters, achieving comparable quality at a fraction of the cost.
The key insight: weight updates during fine-tuning have low intrinsic rank. Instead of updating a full d x d weight matrix W, LoRA decomposes the update into two small matrices.
Original: h = W·x (W is d×d, e.g. 4096×4096)
With LoRA: h = W·x + (B·A)·x where A is r×d, B is d×r
W is frozen. Only A and B are trained. With r=16 and d=4096, trainable params per layer = 2 × 16 × 4096 = 131,072 instead of 16,777,216 — a 128x reduction.
Imagine the frozen weight matrix W as a large wall of bricks. LoRA adds a thin “bypass channel” alongside it: data flows through A (compress to rank r), then through B (expand back to full dimension). The original wall stays untouched.
h = W·x + (alpha/r)·B·A·x| Rank | Trainable Params (per layer pair) | Use Case |
|---|---|---|
| r = 4 | ~33K | Simple tasks, style transfer, single-domain adaptation |
| r = 8 | ~66K | General-purpose fine-tuning (most common default) |
| r = 16 | ~131K | Complex tasks, multi-task learning |
| r = 32 | ~262K | Large domain shift, approaching full FT quality |
Alpha controls the scaling factor of the adapter: scaling = alpha / r. A common rule of thumb is to set alpha = 2 × r (e.g., r=16, alpha=32). Higher alpha means a stronger adapter signal relative to the frozen weights.
LoRA can be applied to any linear layer, but common targets in transformer attention are:
q_proj, v_proj — the original paper's default (best bang for buck)q_proj, k_proj, v_proj, o_proj — all attention projectionsgate_proj, up_proj, down_proj — includes MLP layers (more params, sometimes better quality)QLoRA (Dettmers et al., 2023) takes LoRA further by quantizing the frozen base model to 4 bits while keeping the LoRA adapters in higher precision. Three key innovations:
NormalFloat4 — an information-theoretically optimal 4-bit data type for normally distributed weights. Better than uniform int4 quantization because neural network weights follow a roughly Gaussian distribution.
Quantizes the quantization constants themselves. Each block of 64 weights uses a single fp32 scale factor; double quantization compresses those scale factors with a second round of quantization, saving ~0.37 bits per parameter.
Uses NVIDIA unified memory to page optimizer states to CPU RAM when GPU memory is full. This prevents out-of-memory crashes during gradient spikes without permanently moving data off GPU. Enabled via optim="paged_adamw_32bit" in the Trainer.
Together these techniques let you fine-tune a 65B-parameter model on a single 48GB GPU — or a 7B model on a free Colab T4 (16GB).
HuggingFace's peft library makes LoRA and QLoRA configuration straightforward.
from peft import LoraConfig, get_peft_model, TaskType
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # rank
lora_alpha=32, # scaling factor
lora_dropout=0.05, # regularization
target_modules=["q_proj", "v_proj"],
bias="none",
)
model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()
# trainable params: 6,553,600 || all params: 6,744,682,496 || trainable%: 0.0971from transformers import BitsAndBytesConfig
import torch
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
quantization_config=bnb_config,
device_map="auto",
)| Aspect | Full Fine-Tuning | LoRA (fp16) | QLoRA (4-bit) |
|---|---|---|---|
| VRAM (7B model) | ~56 GB | ~16 GB | ~6 GB |
| Trainable params | 100% | 0.1–1% | 0.1–1% |
| Training speed | Baseline | ~1.5x faster | ~1.2x faster* |
| Quality | Best (ceiling) | 95–100% of full FT | 90–99% of full FT |
| Adapter size | Full model copy | 10–100 MB | 10–100 MB |
| Serving | Load full model | Merge into base or swap adapters | Merge into base or swap adapters |
| Multi-task | Separate model per task | Swap adapters, share base | Swap adapters, share base |
*QLoRA training can be slightly slower than fp16 LoRA because of quantization/dequantization overhead during the forward pass, but the massive VRAM savings usually make it worthwhile.
With LoRA, you can serve one base model + many small adapters for different tasks or customers. Adapter swapping takes milliseconds. This is the foundation of multi-tenant fine-tuning platforms.
nvidia-smi. Document the difference.