Lab Guide

Format: Python script (lab.py)  ·  Time estimate: 45–60 minutes  ·  Prerequisites: GPU access (Colab T4 or better), transformers, peft, bitsandbytes, datasets, trl

Setup

Before starting, ensure your environment is ready:

# Install required packages
pip install transformers peft bitsandbytes datasets trl accelerate torch

# Verify GPU is available
python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, Device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"CPU\"}')"

Exercise 1: Configure LoRA with PEFT

Goal

Load a pre-trained model and attach LoRA adapters. Inspect the trainable parameter count and understand how different configurations affect it.

Steps

1. Load a base model using AutoModelForCausalLM. For this exercise, use a small model that fits in memory:

   from transformers import AutoModelForCausalLM, AutoTokenizer
   
   model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
   tokenizer = AutoTokenizer.from_pretrained(model_name)
   model = AutoModelForCausalLM.from_pretrained(
       model_name, torch_dtype="auto", device_map="auto"
   )

2. Create a LoraConfig with the following settings:
   • r=8, lora_alpha=16
   • target_modules=["q_proj", "v_proj"]
   • lora_dropout=0.05, bias="none"
   • task_type="CAUSAL_LM"
3. Apply the LoRA config using get_peft_model().
4. Call model.print_trainable_parameters() and record the output.
5. Now change the configuration:
   • Increase rank to r=16 — what happens to trainable params?
   • Add k_proj and o_proj to target modules — what happens?
   • Set r=32 with all attention + MLP modules — how close are you to full fine-tuning?

Expected Output

=== LoRA Config: r=8, targets=[q_proj, v_proj] ===
trainable params: 1,572,864 || all params: 1,101,627,392 || trainable%: 0.1428

=== LoRA Config: r=16, targets=[q_proj, v_proj] ===
trainable params: 3,145,728 || all params: 1,103,200,256 || trainable%: 0.2852

=== LoRA Config: r=16, targets=[q_proj, k_proj, v_proj, o_proj] ===
trainable params: 6,291,456 || all params: 1,106,346,112 || trainable%: 0.5689

=== LoRA Config: r=32, targets=[q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj] ===
trainable params: 28,311,552 || all params: 1,128,366,208 || trainable%: 2.5091

Exercise 2: QLoRA Setup with BitsAndBytes

Goal

Load the same model in 4-bit quantization using bitsandbytes and attach LoRA adapters. Compare memory usage with the fp16 version.

Steps

1. Create a BitsAndBytesConfig for 4-bit loading:

   from transformers import BitsAndBytesConfig
   import torch
   
   bnb_config = BitsAndBytesConfig(
       load_in_4bit=True,
       bnb_4bit_quant_type="nf4",
       bnb_4bit_compute_dtype=torch.bfloat16,
       bnb_4bit_use_double_quant=True,
   )

2. Load the model with this config:

   model_4bit = AutoModelForCausalLM.from_pretrained(
       model_name,
       quantization_config=bnb_config,
       device_map="auto",
   )

3. Prepare the model for k-bit training:

   from peft import prepare_model_for_kbit_training
   model_4bit = prepare_model_for_kbit_training(model_4bit)

4. Apply the same LoRA config from Exercise 1 (r=8, q_proj + v_proj).
5. Print trainable parameters and compare with the fp16 version.
6. Check GPU memory usage:

   print(f"GPU memory allocated: {torch.cuda.memory_allocated() / 1024**3:.2f} GB")
   print(f"GPU memory reserved:  {torch.cuda.memory_reserved() / 1024**3:.2f} GB")

Expected Output

=== QLoRA: 4-bit base + LoRA (r=8, q_proj + v_proj) ===
trainable params: 1,572,864 || all params: 578,142,208 || trainable%: 0.2720

GPU memory allocated: 0.82 GB
GPU memory reserved:  1.14 GB

(Compare: fp16 model uses ~2.2 GB for TinyLlama)

Exercise 3: Fine-Tune on a Small Dataset

Goal

Train the QLoRA model on a small instruction-following dataset using the HuggingFace Trainer.

Steps

1. Load a small instruction dataset:

   from datasets import load_dataset
   
   dataset = load_dataset("databricks/databricks-dolly-15k", split="train[:1000]")
   
   def format_instruction(example):
       """Format as a simple instruction-response pair."""
       text = f"### Instruction:\n{example['instruction']}\n\n"
       if example.get("context"):
           text += f"### Context:\n{example['context']}\n\n"
       text += f"### Response:\n{example['response']}"
       return {"text": text}
   
   dataset = dataset.map(format_instruction)

2. Configure the tokenizer for training:

   tokenizer.pad_token = tokenizer.eos_token
   tokenizer.padding_side = "right"

3. Set up SFTTrainer from the trl library:

   from trl import SFTTrainer, SFTConfig
   
   training_args = SFTConfig(
       output_dir="./lora-output",
       num_train_epochs=1,
       per_device_train_batch_size=4,
       gradient_accumulation_steps=4,
       learning_rate=2e-4,
       weight_decay=0.01,
       logging_steps=10,
       save_strategy="epoch",
       optim="paged_adamw_32bit",
       bf16=True,
       max_seq_length=512,
       dataset_text_field="text",
   )
   
   trainer = SFTTrainer(
       model=model_4bit,
       train_dataset=dataset,
       args=training_args,
       tokenizer=tokenizer,
   )

4. Train the model:

   trainer.train()

5. Monitor the training loss — it should decrease steadily over the 1000-example epoch.

Expected Output

{'loss': 2.3456, 'learning_rate': 0.0002, 'epoch': 0.16}
{'loss': 1.8923, 'learning_rate': 0.00018, 'epoch': 0.32}
{'loss': 1.6512, 'learning_rate': 0.00016, 'epoch': 0.48}
{'loss': 1.5234, 'learning_rate': 0.00014, 'epoch': 0.64}
{'loss': 1.4567, 'learning_rate': 0.00012, 'epoch': 0.80}
{'loss': 1.3890, 'learning_rate': 0.0001, 'epoch': 0.96}

Training complete. Total training time: ~5 minutes on T4.

Exercise 4: Merge and Save the Adapter

Goal

Save the trained adapter separately, then merge it into the base model for inference.

Steps

1. Save just the adapter weights:

   # Save adapter only (~few MB)
   model_4bit.save_pretrained("./lora-adapter")
   print("Adapter saved. Check the file size:")

2. List the saved files and note their sizes:

   import os
   for f in os.listdir("./lora-adapter"):
       size = os.path.getsize(f"./lora-adapter/{f}")
       print(f"  {f}: {size / 1024:.1f} KB")

3. Reload the base model in fp16 and merge the adapter:

   from peft import PeftModel
   
   # Reload base model in fp16 for merging
   base_model = AutoModelForCausalLM.from_pretrained(
       model_name, torch_dtype=torch.float16, device_map="auto"
   )
   
   # Load adapter on top
   merged_model = PeftModel.from_pretrained(base_model, "./lora-adapter")
   
   # Merge adapter into base weights
   merged_model = merged_model.merge_and_unload()
   
   # Save the full merged model
   merged_model.save_pretrained("./merged-model")
   tokenizer.save_pretrained("./merged-model")
   print("Merged model saved.")

4. Test inference with the merged model:

   from transformers import pipeline
   
   pipe = pipeline("text-generation", model="./merged-model",
                   tokenizer=tokenizer, max_new_tokens=128)
   
   prompt = "### Instruction:\nExplain what LoRA is in one sentence.\n\n### Response:\n"
   output = pipe(prompt, do_sample=False)
   print(output[0]["generated_text"])

Expected Output

=== Adapter Files ===
  adapter_config.json: 0.5 KB
  adapter_model.safetensors: 6144.0 KB

=== Merged Model Test ===
### Instruction:
Explain what LoRA is in one sentence.

### Response:
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique
that injects small trainable matrices into frozen model layers, allowing
adaptation to new tasks while updating less than 1% of the original parameters.

Finishing Up

Once you have completed all four exercises:

1. Run the full script end-to-end to verify everything executes without errors.
2. Note down the memory usage differences between fp16 LoRA and QLoRA.
3. Compare your implementation with lab_answer.py.

Self-Check

• ✓ Exercise 1: Can you explain how rank and target modules affect the trainable parameter count?
• ✓ Exercise 2: Did QLoRA reduce GPU memory usage compared to fp16? By how much?
• ✓ Exercise 3: Did the training loss decrease over the epoch? What was the final loss?
• ✓ Exercise 4: Is the adapter file significantly smaller than the full model? Can the merged model generate coherent text?
• ✓ Can you explain the difference between saving an adapter vs saving a merged model, and when you would use each?