Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Lessons 20–21 (tokenization and transformer architecture)

1. Pre-training: Learning Language from Raw Text

Every modern LLM begins its life with pre-training — an unsupervised learning stage where the model consumes trillions of tokens and learns to predict the next one. This single objective, next-token prediction (also called causal language modelling), is the foundation upon which all other capabilities are built.

The training objective is mathematically simple: given a sequence of tokens t_1, t_2, ..., t_n, minimise the cross-entropy loss of predicting t_{n+1}. Across trillions of examples, this forces the model to learn grammar, facts, reasoning patterns, coding conventions, and even elements of common sense — all as a side effect of getting better at prediction.

Training Data at Scale

Pre-training datasets are massive and diverse. A typical modern training corpus includes:

Web crawl data (Common Crawl, RefinedWeb): the largest source by volume, providing breadth of knowledge and conversational patterns. This data requires extensive filtering and deduplication to remove low-quality, duplicated, or harmful content.
Books (Books3, Project Gutenberg): long-form text that teaches narrative coherence, extended reasoning, and diverse writing styles.
Code (GitHub repositories, Stack Overflow): programming languages teach the model structured logic, formal syntax, and problem-solving patterns. Research has shown that including code in pre-training improves reasoning performance even on non-code tasks.
Academic and reference text (arXiv, PubMed, Wikipedia): high-quality, factual content that grounds the model's knowledge.

The data mix matters enormously. Llama 2's training data was roughly 90% web data, with the remainder split across code, academic papers, and books. Different mixes produce models with different strengths — more code produces better reasoners; more books produce more coherent long-form writers.

Compute Requirements and Loss Curves

Pre-training is staggeringly expensive. Training Llama 2 70B required approximately 1.7 million GPU-hours on NVIDIA A100 accelerators. Frontier models from Google, OpenAI, and Anthropic use even more compute, with estimates ranging from $10M to over $100M per training run.

During training, the loss (a measure of how wrong the model's predictions are) follows a characteristic pattern. It drops steeply in the first few percent of training as the model learns basic patterns, then follows a smooth power-law decay — each doubling of compute yields a smaller but predictable improvement. Occasional loss spikes indicate training instabilities, often caused by problematic data batches or learning rate issues.

The Chinchilla scaling law (Hoffmann et al., 2022) established that for a given compute budget, you should scale model parameters and training tokens at roughly equal rates. Specifically, the optimal number of training tokens is approximately 20 times the number of parameters. This finding revealed that many early LLMs, including the original GPT-3, were significantly undertrained — they had more parameters than their training data could properly utilise.

2. Supervised Fine-Tuning (SFT): Teaching Instruction Following

A pre-trained model is a powerful text completer, but it is not a useful assistant. If you ask it "What is photosynthesis?", it might respond by listing more questions (because that pattern is common on the web) rather than answering yours. Supervised Fine-Tuning bridges this gap.

In SFT, the model is trained on curated (instruction, response) pairs written by human demonstrators. These examples teach the model the chat format — that when a user asks a question, the expected behaviour is to provide a helpful, well-structured answer. The training objective is still next-token prediction, but the loss is computed only on the assistant's response tokens, not the user's prompt.

SFT datasets are much smaller than pre-training data — typically 10,000 to 100,000 high-quality demonstrations. The compute cost is correspondingly lower: hours to days rather than weeks to months. Despite this relatively small investment, SFT produces a dramatic change in model behaviour. The model shifts from completing text to following instructions, adopting a helpful and structured communication style.

Chat Templates

SFT also establishes the chat format that the model will use at inference time. Models learn to distinguish between system, user, and assistant roles, and to generate responses only in the assistant role. This is why API calls use structured message arrays rather than raw text strings.

3. RLHF: Aligning with Human Preferences

SFT produces a model that follows instructions, but it does not guarantee that the model's outputs are good. It might generate plausible-sounding but incorrect answers, produce harmful content when asked, or be sycophantic rather than truthful. Reinforcement Learning from Human Feedback (RLHF) addresses these shortcomings by optimising the model to produce outputs that humans actually prefer.

Step 1: Collecting Preference Data

Human annotators are presented with a prompt and two candidate responses (both generated by the SFT model). They indicate which response is better, considering factors like helpfulness, accuracy, and safety. This produces a dataset of preference pairs: (prompt, chosen response, rejected response).

Step 2: Training a Reward Model

A separate neural network — the reward model (RM) — is trained on the preference data. Given a (prompt, response) pair, the RM outputs a scalar score representing how good the response is. The RM is trained so that chosen responses receive higher scores than rejected ones. This model essentially distils human judgment into a differentiable function.

Step 3: Policy Optimization with PPO

Proximal Policy Optimization (PPO) is the reinforcement learning algorithm used to update the language model. The model generates responses, the reward model scores them, and the model's parameters are updated to increase the probability of high-scoring responses. Crucially, a KL divergence penalty prevents the model from drifting too far from the SFT baseline. Without this constraint, the model would learn to exploit quirks in the reward model (reward hacking) rather than genuinely improving quality.

Reward Hacking

If the KL penalty is too weak, models can discover degenerate strategies that score high with the reward model but produce gibberish or repetitive text. If it is too strong, the model barely changes from the SFT version. Finding the right balance is one of the key challenges of RLHF.

4. Constitutional AI: Anthropic's Approach

Anthropic developed Constitutional AI (CAI) as an alternative to purely human-driven RLHF. The core insight is that instead of relying on human annotators for every safety judgment, you can define a set of principles — a "constitution" — and have the model critique and revise its own outputs against those principles.

The CAI process works in two phases:

Critique and revision (supervised): The model generates a response, then is prompted to evaluate whether it violates any constitutional principle. If it does, the model revises its response. The revised responses become SFT training data.
RLAIF (RL from AI Feedback): Instead of human preference labels, the model itself (or a separate model) generates preference judgments guided by the constitution. These AI-generated preferences are used to train a reward model, which is then used for PPO just like standard RLHF.

CAI reduces the burden on human annotators, makes the alignment criteria explicit and auditable, and scales better than pure RLHF. Anthropic's Claude models are trained using this approach, with a constitution that emphasises helpfulness, harmlessness, and honesty.

5. DPO: A Simpler Alternative

Direct Preference Optimization (DPO), introduced by Rafailov et al. in 2023, offers a fundamentally simpler approach to preference-based alignment. The key insight is that the optimal policy under the RLHF objective can be expressed in closed form, eliminating the need for a separate reward model and the complexity of PPO.

DPO directly optimises the language model on preference pairs using a classification-like loss function. For each preference pair, it increases the likelihood of the chosen response and decreases the likelihood of the rejected response, with an implicit KL constraint built into the loss.

The advantages of DPO over RLHF are significant:

Simplicity: no reward model to train, no RL loop to stabilise
Stability: standard supervised training, no PPO hyperparameter tuning
Cost: requires only one model instead of two (policy + reward model)
Accessibility: can be done with standard fine-tuning infrastructure

DPO has been widely adopted in the open-source community. Llama 3, Zephyr, and many other models use DPO as their alignment method. For practitioners doing their own fine-tuning (covered in Module 5), DPO is typically the recommended approach.

6. The Complete Pipeline

Stage	Objective	Data	Cost
Pre-training	Next-token prediction	1–15T tokens (web, books, code)	$10M–$100M+
SFT	Instruction following	10K–100K demonstrations	$10K–$100K
RLHF / CAI / DPO	Alignment with preferences	50K–500K preference pairs	$50K–$500K

As a practitioner, you will almost never do pre-training from scratch. You will frequently do SFT (Module 5) and occasionally DPO. Understanding the full pipeline helps you reason about model behaviour: why base models are incoherent as assistants, why SFT models sometimes hallucinate confidently, and why aligned models sometimes refuse harmless requests.