Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Lesson 25 (fine-tuning fundamentals), basic PyTorch familiarity

1. Why Parameter-Efficient Fine-Tuning?

Full fine-tuning updates every parameter in a pre-trained model. For a 7-billion parameter model stored in fp16, the weights alone occupy 14 GB. Add optimizer states (AdamW stores two additional copies of every parameter) and gradients, and you need roughly 56 GB of VRAM just for training — well beyond what a single consumer GPU can provide.

This creates a practical problem: most teams and individual practitioners cannot afford multi-GPU clusters for every fine-tuning experiment. Parameter-Efficient Fine-Tuning (PEFT) methods solve this by freezing the original model weights and training only a small set of additional parameters. The frozen weights provide the model's general knowledge, while the trainable parameters learn the task-specific adaptation.

Among PEFT methods, LoRA has emerged as the dominant approach due to its simplicity, effectiveness, and the fact that trained adapters can be merged back into the base model at inference time with zero additional latency.

2. LoRA: Low-Rank Adaptation

LoRA was introduced by Hu et al. (2021) in the paper "LoRA: Low-Rank Adaptation of Large Language Models." The core observation is that the weight updates learned during fine-tuning occupy a low-dimensional subspace — that is, the change matrix (delta W) has low intrinsic rank even though the full weight matrix is high-dimensional.

The Decomposition

For a pre-trained weight matrix W of shape d x k, LoRA represents the fine-tuning update as:

delta W = B · A

where A has shape r x k and B has shape d x r, with the rank r << min(d, k). The forward pass becomes:

# Conceptual forward pass with LoRA
h = W @ x + (alpha / r) * (B @ A) @ x

The original weight matrix W is frozen — it receives no gradient updates. Only the small matrices A and B are trained. For a typical transformer layer where d = k = 4096 and r = 16, the trainable parameters per adapted layer drop from 16.8 million to 131,072 — a 128x reduction.

Initialization

A is initialized with a random Gaussian distribution, while B is initialized to all zeros. This means the LoRA adapter starts as a no-op: at the beginning of training, B · A = 0, so the model behaves exactly like the pre-trained base. This is important because it ensures training starts from a known-good state rather than a random perturbation.

The Alpha Parameter

The scaling factor alpha / r controls how much influence the adapter has relative to the frozen weights. When you increase alpha, the adapter's contribution is amplified. A common practice is to set alpha = 2 * r, giving a scaling factor of 2.0. However, this interacts with your learning rate: a higher alpha with a lower learning rate can produce similar results to a lower alpha with a higher learning rate. The key is that alpha gives you an additional knob to control training dynamics without changing the number of trainable parameters.

Which Layers to Adapt

LoRA can be applied to any linear layer in the transformer, but not all layers benefit equally. The original paper found that adapting the attention projection matrices — specifically q_proj (query) and v_proj (value) — provided the best trade-off between parameter count and quality. This makes intuitive sense: the attention mechanism is where the model decides what information to focus on, and task-specific adaptation often involves changing these attention patterns.

In practice, many practitioners also include k_proj and o_proj (all four attention projections), and some extend to the MLP layers (gate_proj, up_proj, down_proj in Llama-style architectures). Adding more target modules increases the trainable parameter count but can improve quality for complex tasks that require changing both what the model attends to and how it transforms that information.

Choosing the Rank

The rank r determines the expressiveness of the adapter. Common choices are:

r = 4: Minimal adaptation. Suitable for simple tasks like formatting changes or style transfer where the base model already has the required knowledge.
r = 8: The most common default. Works well for instruction tuning, classification, and moderate domain adaptation.
r = 16: Good for tasks requiring more substantial behavioral changes or when adapting to specialized domains (medical, legal, code).
r = 32–64: Approaching the capacity of full fine-tuning. Useful when you need maximum quality and have the VRAM budget, but still want the benefits of adapter modularity.

Higher rank is not always better. If the rank exceeds the intrinsic dimensionality of your task's update, the extra parameters can lead to overfitting, especially on small datasets. Start with r = 8 and increase only if validation metrics plateau.

3. QLoRA: Quantization Meets LoRA

QLoRA (Dettmers et al., 2023) extends LoRA with a crucial addition: the frozen base model is quantized to 4 bits before LoRA adapters are applied. This reduces the memory footprint of the frozen weights by roughly 4x compared to fp16, enabling fine-tuning of large models on hardware that would otherwise be insufficient.

NormalFloat4 (NF4)

Standard 4-bit integer quantization divides the value range into 16 equally spaced bins. But neural network weights are not uniformly distributed — they follow an approximately Gaussian distribution centered around zero. NF4 is an information-theoretically optimal quantization scheme for normally distributed data: the 16 quantization levels are spaced so that each bin contains an equal number of values from the expected distribution. This results in significantly lower quantization error compared to naive int4.

Double Quantization

Quantization requires storing scale factors (and sometimes zero-points) for each block of weights, typically one fp32 constant per 64 weights. For a 7B-parameter model, these constants alone add up to ~0.5 GB. Double quantization applies a second round of quantization to these constants, compressing them from fp32 to fp8 with their own block-level constants. This saves approximately 0.37 bits per parameter, which translates to roughly 325 MB for a 7B model.

Paged Optimizers

Even with 4-bit base weights, optimizer states for the LoRA parameters are stored in fp32 and can cause out-of-memory errors during gradient accumulation spikes. QLoRA uses NVIDIA unified memory (managed by the CUDA driver) to automatically page optimizer states between GPU and CPU memory. When GPU memory is exhausted, pages are evicted to CPU RAM; when needed again, they are paged back in. This is configured by using paged_adamw_32bit as the optimizer in the HuggingFace Trainer.

Putting It Together

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

# Step 1: Load model in 4-bit
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

# Step 2: Prepare for k-bit training (handles gradient checkpointing, etc.)
model = prepare_model_for_kbit_training(model)

# Step 3: Apply LoRA
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 6,553,600 || all params: 3,540,389,888 || trainable%: 0.1851

4. Merging Adapters

One of LoRA's most practical advantages is that trained adapters can be merged back into the base model at inference time. Since the adapter computes delta W = B · A, you simply add this to the original weight matrix: W_new = W + (alpha/r) · B · A. The merged model has the exact same architecture and inference speed as the original — no adapter overhead at serving time.

# Merge and save
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")

Alternatively, you can save just the adapter weights (~10–100 MB) and load them on top of the base model at inference time. This is useful when you maintain multiple task-specific adapters that share a single base model — a pattern sometimes called adapter serving.

5. When to Use What

Scenario	Recommended Approach
Small dataset (<1K examples), simple task	LoRA with r=4–8, q_proj + v_proj only
Medium dataset, instruction tuning	LoRA with r=8–16, all attention projections
Large dataset, significant domain shift	LoRA with r=16–32, attention + MLP layers
Limited GPU memory (T4, RTX 3060)	QLoRA with any of the above LoRA configs
Maximum quality, budget available	Full fine-tuning (but try LoRA first)
Multi-task / multi-tenant serving	LoRA adapters with shared base model