A ground-up look at the technology powering modern AI assistants — from probability distributions to transformer architectures.
At its core, a language model assigns a probability distribution over the next token given all previous tokens. Training on trillions of tokens from books, code, and web text compresses world knowledge into a function that can continue any sequence plausibly.
Given the prefix "The capital of France is", the model assigns probabilities like:
| Continuation token | Approximate probability |
|---|---|
| Paris | ~94% |
| Lyon | ~2% |
| a | ~1.5% |
| (anything else) | <2.5% combined |
Chaining this prediction step-by-step produces coherent text — or code, or JSON, or poetry.
Models don't read characters or words — they read tokens, sub-word pieces produced by a learned vocabulary (typically 50–100 K entries).
hello → 1 tokentokenization → 2 tokenssupercalifragilistic → ~6 tokensimport tiktoken
enc = tiktoken.get_encoding("cl100k_base") # Claude / GPT-4 family proxy
text = "Hello, how many tokens is this sentence?"
tokens = enc.encode(text)
print(f"{len(tokens)} tokens: {tokens}")
# → 9 tokens: [9906, 11, 1268, 1690, 11460, 374, 420, 11914, 30]
Every modern LLM is built on the transformer architecture. Here is the data flow at inference time:
Input text → tokenise → embedding lookup (token + position) → stack of attention + FFN layers → logits over vocabulary → sample next token → repeat
For each token, attention asks: "which other tokens in this context are most relevant for predicting what comes next?" and weights their representations accordingly.
Claude, GPT-4, Llama, Mistral — all decoder-only. They use causal masking so each position can only attend to earlier positions, making autoregressive generation efficient.
The context window is the maximum number of tokens a model can process in a single call — both input and output combined.
| Model | Context window | Approx. pages |
|---|---|---|
| Claude Haiku 3.5 | 200 K tokens | ~500 pages |
| Claude Sonnet 4 | 200 K tokens | ~500 pages |
| GPT-4o | 128 K tokens | ~320 pages |
| Gemini 1.5 Pro | 1 M tokens | ~2 500 pages |
Input tokens are priced per-million. Sending a 100 K-token document on every turn costs money — cache or chunk when possible.
Research shows LLMs recall information placed at the beginning and end of long contexts better than content in the middle. Put critical instructions at the top or bottom.
Temperature scales the logit distribution before softmax. Low = peaked distribution (safe). High = flat distribution (creative/risky).
Always picks the highest-probability token. Deterministic. Best for: code, structured data, factual Q&A.
Balanced creativity and coherence. Best for: summarisation, drafting, general chat.
High diversity, occasional incoherence. Best for: brainstorming, creative writing, generating varied options.
Keep the smallest token set whose cumulative probability ≥ p. top_p=0.9 is a popular default. Don't set both temperature and top_p simultaneously.
Live walkthrough of the concepts from this lesson using demo/tokenizer_demo.py.
demo/tokenizer_demo.py in your editortemperature parameter and re-run the same creative prompt at 0, 0.7, and 1.2cl100k_base vs p50k_base) exist and when you'd choose each.