Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Basic Python, no prior ML knowledge required

1. A Brief History: From N-Grams to Transformers

To understand modern large language models, it helps to understand what came before them.

N-Gram Models (1990s–2000s)

The earliest statistical language models were n-gram models. An n-gram model estimates the probability of a word given the previous n-1 words. A bigram model (n=2) predicts the next word from only the current word; a trigram model uses the previous two words.

These models were useful but fundamentally limited: they could only look back a fixed, small number of words, and they required storing every possible n-gram sequence seen in the training corpus. For large vocabularies, the storage requirements became impractical, and they had no ability to generalise — if a sequence hadn't appeared in training data, the model had no way to handle it.

Recurrent Neural Networks (2010s)

Recurrent Neural Networks (RNNs), and later Long Short-Term Memory networks (LSTMs), addressed the fixed-window limitation. These architectures processed sequences step by step, maintaining a hidden state that theoretically could carry information across arbitrarily long sequences.

In practice, however, RNNs suffered from the vanishing gradient problem: information about tokens many steps back in the sequence would fade away during training, making it difficult for the model to learn long-range dependencies. LSTMs improved this with gating mechanisms, but they were still sequential — each token had to be processed one at a time, making training slow and parallelisation difficult.

The Transformer Revolution (2017–Present)

In 2017, the paper Attention Is All You Need (Vaswani et al.) introduced the transformer architecture, which discarded recurrence entirely. Instead of processing tokens sequentially, the transformer allows every token to directly attend to every other token in the sequence — simultaneously. This breakthrough enabled:

Parallelisation: training on GPUs became dramatically more efficient
Long-range dependencies: any token can directly relate to any other token, regardless of distance
Scale: the architecture scales predictably with more data and more parameters

The GPT series (GPT-1 in 2018, GPT-2 in 2019, GPT-3 in 2020) demonstrated that scaling transformer decoders trained on vast amounts of text produced models with increasingly remarkable capabilities. Claude, GPT-4, Llama, and Gemini are all descendants of this lineage.

2. Attention: The Core Innovation

The attention mechanism is what makes transformers powerful. Here is the intuition without the mathematics.

Imagine you are reading the sentence: "The animal didn't cross the street because it was too tired."

When you process the word "it", you need to resolve what "it" refers to — the animal, or the street? You naturally pay more attention to "animal" and "tired" than to "street" and "cross". Attention is the mechanism that allows the model to do the same thing computationally.

How Attention Works

Each token is transformed into three vectors:

Query (Q): "What am I looking for?"
Key (K): "What do I contain?"
Value (V): "What information should I pass on if I'm attended to?"

The attention score between two tokens is computed by taking the dot product of one token's Query with another token's Key, then normalising across all tokens using a softmax. The result is a set of attention weights — a probability distribution over all tokens. The model then computes a weighted sum of the Value vectors to produce the output for each token position.

Multi-head attention runs this process multiple times in parallel with different learned projections, allowing the model to simultaneously attend to different kinds of relationships (syntactic, semantic, positional, etc.).

Causal (Decoder) vs. Bidirectional (Encoder) Attention

GPT-style models use causal masking: when processing token t, the model can only attend to tokens 0 through t-1. This enforces the constraint that generation is left-to-right — the model cannot "look ahead". This is why generative models are called decoders.

BERT-style models use bidirectional attention: every token can attend to every other token. This is better for tasks like classification and information retrieval, but means the model cannot be used for generation directly.

3. The GPT Architecture Overview

A GPT-style model is a stack of transformer decoder blocks. Here is what happens to your text when you send it to the API:

Tokenisation: Your input string is converted to a sequence of integer token IDs using a tokeniser (e.g., byte-pair encoding / BPE). The model never sees raw characters.
Embedding: Each token ID is mapped to a high-dimensional vector (e.g., 4096 dimensions for a mid-size model). These vectors are learned during training; semantically similar tokens end up near each other in this space.
Positional encoding: Because attention has no inherent notion of order, positional information is added to the embeddings — either as fixed sinusoidal encodings (original transformer) or learned positional embeddings (most modern models).
Transformer blocks (repeated N times):
- Multi-head causal self-attention
- Layer normalisation
- Feed-forward network (two linear layers with a non-linearity between them)
- Residual connections (skip connections that help with training deep networks)
Output head: The final token representation is projected back to vocabulary size and passed through a softmax to produce a probability distribution over all possible next tokens.
Sampling: A token is drawn from this distribution (controlled by temperature, top-p, etc.), appended to the sequence, and the process repeats until a stop condition is met.

4. Emergent Abilities

One of the most surprising findings in LLM research is the phenomenon of emergent abilities — capabilities that appear abruptly as model scale increases, rather than improving gradually.

Examples of emergent abilities include:

Few-shot learning: the ability to perform tasks from just a few examples in the prompt, with no gradient updates
Chain-of-thought reasoning: with sufficiently large models, simply prompting them to "think step by step" dramatically improves performance on multi-step reasoning tasks
In-context learning: the model can adapt its behaviour based on examples provided in the context window

Why do these abilities emerge? This is still an active research question. One hypothesis is that these capabilities require many sub-skills to be learned simultaneously, and they only become coherent once the model is large enough to have learned all of them.

This has an important practical implication: techniques that work on GPT-4 class models may not work on smaller models, and vice versa. Always test your prompting strategies on the specific model and size you intend to deploy.

5. Scaling Laws: A Teaser

In 2020, researchers at OpenAI published the paper Scaling Laws for Neural Language Models (Kaplan et al.), which showed that LLM performance follows smooth, predictable power laws as a function of:

Model size (number of parameters)
Dataset size (number of training tokens)
Compute (FLOPs used during training)

This meant that — within the transformer architecture — you could predict how good a model would be before training it, simply based on how much you were willing to spend. This predictability drove the investment in ever-larger models.

The follow-up Chinchilla paper (Hoffmann et al., 2022) refined these laws and showed that many earlier large models were significantly undertrained relative to their size. The optimal training approach is to scale the dataset size at roughly the same rate as the model size.

For practitioners, the key takeaway from scaling laws is: the largest model you can afford is often the right choice for capability, but not always for cost-efficiency. Smaller fine-tuned models can match or exceed larger general-purpose models on narrow tasks.

6. Key Concepts Summary

Concept	What it means	Why it matters
Token	Atomic unit of text (subword)	Determines cost and context usage
Context window	Max tokens in a single call (input + output)	Hard limit on what the model can "see"
Temperature	Controls randomness of sampling	Tune for task type (0 for facts, ~1 for creativity)
Attention	Mechanism for relating tokens to each other	Core of why transformers work
Emergent ability	Capability that appears only above a scale threshold	Explains why model choice matters
Scaling laws	Performance predictably follows compute/data/size	Guides model selection and training strategy