Lesson 22 · Slide 1
Pre-training & RLHF
How LLMs go from random weights to helpful assistants — the full training pipeline from raw text prediction to alignment with human preferences.
Learning Objectives
- Explain the pre-training objective (next-token prediction) and the data/compute it requires
- Describe Supervised Fine-Tuning (SFT) and why it turns a base model into an instruction follower
- Understand RLHF: reward models, PPO, and preference data
- Compare Constitutional AI and DPO as alignment alternatives
- Trace the full pipeline: Pre-train → SFT → RLHF/DPO
Lesson 22 · Slide 2
Stage 1: Pre-training
Pre-training is where models learn language, facts, reasoning patterns, and code — all from a single objective: predict the next token.
The Objective
Given a sequence of tokens t_1, t_2, ..., t_n, minimise the cross-entropy loss of predicting t_{n+1}. This deceptively simple objective, applied at scale, produces emergent capabilities.
Training Data
| Source | Examples | Contribution |
|---|
| Web crawl | Common Crawl, RefinedWeb | Breadth of knowledge, conversational patterns |
| Books | Books3, Project Gutenberg | Long-form reasoning, narrative coherence |
| Code | GitHub, Stack Overflow | Logical reasoning, structured output |
| Academic | arXiv, PubMed, Wikipedia | Factual accuracy, technical depth |
Modern pre-training runs use 1–15 trillion tokens. Data quality and deduplication matter as much as quantity.
Notes: Emphasise that the model sees no labels, no instructions, no feedback — just raw text. Everything it learns comes from patterns in the data.
Lesson 22 · Slide 3
Pre-training: Compute & Loss Curves
Pre-training is the most expensive stage. Training a frontier model costs $10M–$100M+ in compute alone.
Compute Requirements
- Thousands of GPUs (A100/H100) for weeks to months
- Llama 2 70B: ~1.7M GPU-hours on A100s
- Chinchilla scaling: tokens ≈ 20 × parameters
- Distributed across data/tensor/pipeline parallelism
Loss Curves
- Loss drops steeply early, then flattens into a power law decay
- Loss spikes indicate instability — learning rate resets may be needed
- Final loss predicts downstream performance (scaling laws)
- Overfitting is rare at this scale — models are typically undertrained
Chinchilla Scaling Law
For a given compute budget, the optimal strategy is to scale model size and dataset size equally. A 70B model should see ~1.4T tokens. Many early LLMs (GPT-3, PaLM) were undertrained by this criterion.
Notes: The lab will have students plot actual loss curves from published training runs to see the power-law pattern firsthand.
Lesson 22 · Slide 4
Stage 2: Supervised Fine-Tuning (SFT)
A pre-trained model can continue text, but it does not follow instructions. SFT teaches the model to be a helpful assistant by training on curated (instruction, response) pairs.
What Changes
- Training data: thousands to hundreds of thousands of human-written demonstrations
- Format: structured chat templates (
system / user / assistant turns)
- Objective: still next-token prediction, but only on the assistant response tokens
- Compute: much cheaper than pre-training — hours to days, not months
Before vs After SFT
| Prompt | Base Model | After SFT |
|---|
| "What is photosynthesis?" | "What is cellular respiration? What is..." (continues listing questions) | "Photosynthesis is the process by which plants convert sunlight, water, and CO2 into glucose and oxygen..." |
SFT is necessary but not sufficient. The model can now follow instructions but may still produce harmful, dishonest, or unhelpful outputs.
Notes: Point out that the base model completes text plausibly (it was trained on web pages full of questions), but SFT teaches it to answer rather than continue.
Lesson 22 · Slide 5
Stage 3: RLHF
Reinforcement Learning from Human Feedback aligns the model with human preferences — making it not just capable, but helpful, harmless, and honest.
The RLHF Pipeline
- Collect preference data: Present humans with two model responses to the same prompt. They choose which is better.
- Train a reward model (RM): A separate model learns to predict the human preference score for any (prompt, response) pair.
- Optimise with PPO: Use Proximal Policy Optimization to update the language model to maximise reward while staying close to the SFT model (KL penalty).
Why KL Divergence Matters
Without a KL penalty, the model would reward hack — finding degenerate outputs that score high with the reward model but are incoherent to humans. The KL constraint keeps the model close to the SFT baseline.
Notes: RLHF was the key innovation behind ChatGPT. The base GPT-3.5 model existed before, but RLHF made it conversational and aligned enough for public release.
Lesson 22 · Slide 6
Constitutional AI & DPO
RLHF is powerful but expensive and complex. Two important alternatives have emerged.
Constitutional AI (Anthropic)
- Instead of human labellers for every comparison, define a set of principles (a "constitution")
- The model critiques and revises its own outputs against these principles (RLAIF — RL from AI Feedback)
- Reduces dependence on human annotators for safety-related judgments
- Used to train Claude — principles include helpfulness, harmlessness, and honesty
DPO (Direct Preference Optimization)
- Eliminates the separate reward model entirely
- Directly optimises the language model on preference pairs using a closed-form loss function
- Simpler, more stable, cheaper than PPO-based RLHF
- Increasingly popular: used in Llama 3, Zephyr, and many open-source models
Both approaches use preference data, but they differ in how that data is generated (human vs AI) and how the model is updated (RL vs supervised loss).
Lesson 22 · Slide 7
The Full Training Pipeline
Every modern instruction-following LLM goes through some variant of this pipeline:
Stage 1: Pre-training (trillions of tokens, months of compute) → base model that can predict text
Stage 2: SFT (thousands of demonstrations, days of compute) → instruction-following model
Stage 3: Alignment (RLHF / CAI / DPO with preference data) → helpful, harmless, honest assistant
Cost Breakdown
| Stage | Data | Compute | Outcome |
|---|
| Pre-training | 1–15T tokens (web, books, code) | $10M–$100M+ | World knowledge, language competence |
| SFT | 10K–100K demonstrations | $10K–$100K | Instruction following, chat format |
| RLHF / DPO | 50K–500K preference pairs | $50K–$500K | Alignment, safety, quality |
Pre-training is 90%+ of the total cost, but post-training (SFT + alignment) is what makes the model usable.
Notes: Reinforce that you can do SFT and DPO yourself (Module 5 covers this), but pre-training from scratch is only done by well-funded labs.
Lesson 22 · Slide 8
Lab & Homework
Lab Exercises
- Training Loss Curves — plot and analyse published loss curves from Llama 2 and Pythia to observe power-law decay and training instabilities
- Base vs Instruct — compare outputs from a base model and its instruct variant on identical prompts to see the effect of SFT + RLHF
- Preference Data Analysis — explore the Anthropic HH-RLHF dataset: examine preference pairs, identify patterns in what humans prefer
Homework Tasks
- Map the full training pipeline for a published model (Llama 3, Mistral, or Claude). Document each stage, data sources, and alignment method used.
- Write 10 preference pairs for a domain of your choice. For each pair, explain why one response is preferred.
- Compare Constitutional AI and DPO: write 200 words on when you would choose each approach and why.