Lab Guide

Format: Jupyter notebook (lab.ipynb)  ·  Time estimate: 45–60 minutes  ·  Prerequisites: Python 3.9+, matplotlib, numpy, datasets, anthropic, python-dotenv

Setup

Before starting, ensure your environment is ready:

# From the project root
pip install matplotlib numpy datasets anthropic python-dotenv

# Create a .env file in the project root with your API key
echo "ANTHROPIC_API_KEY=your_key_here" > .env

Verify setup by running the first notebook cell — it should import all libraries without errors.

Exercise 1: Training Loss Curves

Goal

Visualise and analyse pre-training loss curves to understand how models learn over time and observe the power-law decay pattern described in scaling laws research.

Steps

1. The notebook provides pre-extracted loss data from published training runs (Llama 2 and Pythia model suites). Load this data using the provided helper function.
2. Plot the loss curve for a single model (Pythia-410M) across all training steps. Use a log-log scale to reveal the power-law relationship.
3. Overlay loss curves for models of different sizes (Pythia-70M, 160M, 410M, 1B, 2.8B) on the same plot. Observe how larger models achieve lower loss at every point in training.
4. Identify and annotate any loss spikes visible in the curves. These indicate training instabilities.
5. Fit a power-law curve (L = a * N^(-b) + c) to one of the loss curves using scipy.optimize.curve_fit. Compare the fit to the actual data.

Expected Output

=== Pythia-410M Loss Summary ===
Initial loss (step 1000):    3.82
Final loss (step 143000):    2.67
Total reduction:             30.1%

=== Power-Law Fit ===
L = 5.41 * N^(-0.078) + 2.51
R-squared: 0.994

=== Model Size vs Final Loss ===
Pythia-70M:   3.12
Pythia-160M:  2.88
Pythia-410M:  2.67
Pythia-1B:    2.48
Pythia-2.8B:  2.30

Key Observations to Note

• Loss drops steeply in early training, then follows a smooth power-law decay.
• Larger models consistently achieve lower loss — but with diminishing returns per parameter increase.
• The log-log plot should appear roughly linear, confirming the power-law relationship.
• Loss spikes are rare but visible — these correspond to training instabilities that required intervention in the original runs.

Exercise 2: Base vs Instruct Model Comparison

Goal

Directly observe the behavioural difference between a base (pre-trained only) model and an instruct (SFT + RLHF) model to understand what post-training accomplishes.

Steps

1. Define a set of 5 test prompts that span different tasks:
   • A factual question: "What is photosynthesis?"
   • An instruction: "Summarise the following text in 2 sentences: [paste a paragraph]"
   • A safety-sensitive query: "How do I pick a lock?"
   • A coding task: "Write a Python function to reverse a linked list."
   • A conversational exchange: "I'm feeling stressed about my exam tomorrow. Any advice?"
2. Use the Claude API to get responses from claude-3-5-haiku-20241022 (instruct model). Record each response.
3. The notebook provides pre-recorded outputs from a base model (Llama 2 7B base, via Hugging Face) for the same prompts. These are included because base models are not available through the Claude API.
4. Create a side-by-side comparison table showing the base model output and the instruct model output for each prompt.
5. For each pair, write a one-sentence annotation explaining what the instruct model does differently and why.

Expected Output

=== Prompt: "What is photosynthesis?" ===

--- Base Model (Llama 2 7B) ---
"What is photosynthesis?\nWhat is cellular respiration?\nWhat is
the relationship between photosynthesis and cellular respiration?\n
These are the questions that we will be exploring in this unit..."

--- Instruct Model (Claude Haiku) ---
"Photosynthesis is the biological process by which plants, algae,
and some bacteria convert light energy into chemical energy stored
in glucose. The process takes place primarily in the chloroplasts
of plant cells, using sunlight, water (H2O), and carbon dioxide
(CO2) to produce glucose (C6H12O6) and oxygen (O2) as a byproduct."

--- Annotation ---
The base model continues the text as if it's a textbook page (listing
more questions), while the instruct model directly answers the question
with a structured, informative response.

Key Observations to Note

• Base models complete text plausibly but rarely answer questions directly.
• Instruct models structure their responses, stay on topic, and address the user's intent.
• On safety-sensitive prompts, the instruct model will decline or add caveats, while the base model has no such guardrails.
• The instruct model adopts a consistent "assistant" persona; the base model has no persona.

Exercise 3: Preference Data Analysis

Goal

Explore a real preference dataset used for RLHF to understand the structure of preference data and identify patterns in what humans prefer.

Steps

1. Load the Anthropic HH-RLHF dataset from Hugging Face:

   from datasets import load_dataset
   
   ds = load_dataset("Anthropic/hh-rlhf", split="train[:500]")
   print(f"Loaded {len(ds)} preference pairs")
   print(f"Columns: {ds.column_names}")
   print(f"\nExample keys: {list(ds[0].keys())}")

2. Examine 10 random preference pairs. For each pair, the dataset contains a chosen and a rejected response. Print both side by side.
3. For each of the 10 pairs, categorise why the chosen response was preferred. Use these categories:
   • More helpful: provides better, more complete information
   • More harmless: avoids harmful, biased, or dangerous content
   • More honest: more accurate, acknowledges uncertainty
   • Better structured: clearer formatting, organisation
4. Compute basic statistics on the dataset:
   • Average length (in characters) of chosen vs rejected responses
   • Distribution of conversation turns in the preference pairs
   • Most common opening patterns in chosen vs rejected responses
5. Create a bar chart showing the length distribution of chosen vs rejected responses. Is there a length bias?

Expected Output

=== Dataset Overview ===
Loaded 500 preference pairs
Columns: ['chosen', 'rejected']

=== Length Statistics ===
Chosen responses  - Mean: 892 chars, Median: 734 chars
Rejected responses - Mean: 651 chars, Median: 523 chars
Length ratio (chosen/rejected): 1.37

=== Preference Categories (10 samples) ===
More helpful:     6/10
More harmless:    2/10
More honest:      1/10
Better structured: 1/10

=== Key Finding ===
Chosen responses are on average 37% longer than rejected ones.
This "length bias" is a known issue in RLHF preference data.

Key Observations to Note

• Chosen responses tend to be longer — this is a well-documented bias in human preference data that can cause RLHF-trained models to be overly verbose.
• Most preferences in this dataset relate to helpfulness rather than safety.
• The dataset format is simple: just chosen and rejected text. No explicit scores or rubrics.
• Some preference pairs are ambiguous — reasonable people might disagree on which is better.

Finishing Up

Once you have completed all three exercises:

1. Run all cells from top to bottom to verify everything executes without errors (Kernel → Restart & Run All).
2. Save the notebook.
3. Compare your implementation with lab_answer.ipynb — focus on your loss curve analysis and preference data insights.

Self-Check

• ✓ Exercise 1: Can you explain what a power-law loss curve implies about the relationship between compute and performance?
• ✓ Exercise 1: Did your power-law fit achieve R-squared > 0.99?
• ✓ Exercise 2: Can you articulate at least 3 specific behavioural differences between base and instruct models?
• ✓ Exercise 2: Did the safety-sensitive prompt produce notably different responses?
• ✓ Exercise 3: Did you identify the length bias in the preference data?
• ✓ Exercise 3: Can you explain why length bias is problematic for RLHF training?