Lesson 20 · Slide 1
Tokenization Deep Dive
How text becomes numbers — the critical first step in every language model pipeline. Tokenization determines what a model can see, how efficiently it processes language, and how much you pay per API call.
Learning Objectives
- Implement the BPE (Byte Pair Encoding) algorithm from scratch
- Compare BPE, WordPiece, Unigram, and SentencePiece approaches
- Understand vocabulary construction and its impact on model capability
- Identify special tokens and their roles in different model families
- Measure multilingual token efficiency and its cost implications
Notes: Module 4 shifts from using LLMs as tools to understanding their internals. Tokenization is the foundation — everything else in the architecture operates on tokens.
Lesson 20 · Slide 2
Why Tokenization Matters
Models never see raw text. Every string is converted into a sequence of integer IDs before any computation. The choice of tokenizer affects vocabulary coverage, sequence length, multilingual fairness, and cost.
Character-Level
- Vocabulary: ~256 entries (bytes)
- Sequences are very long
- No out-of-vocabulary words
- Too slow for practical LLMs
Word-Level
- Vocabulary: 100K+ entries
- Short sequences
- Huge OOV (out-of-vocabulary) problem
- Cannot handle misspellings or new words
Subword tokenization sits in the sweet spot: vocabularies of 30K–100K entries that can represent any text by composing smaller pieces, keeping sequences manageable while eliminating the OOV problem entirely.
Lesson 20 · Slide 3
BPE: Byte Pair Encoding Step by Step
BPE is the most widely used subword algorithm. It was adapted from a data compression technique (Gage, 1994) for NLP by Sennrich et al. (2016).
The Algorithm
- Initialize: Start with a vocabulary of all individual characters (or bytes) in the corpus
- Count pairs: Find the most frequent adjacent pair of tokens in the training data
- Merge: Replace every occurrence of that pair with a new single token
- Repeat steps 2–3 until the vocabulary reaches the desired size
Corpus: "low low low low low lower lower newest newest widest"
Step 0 vocabulary: {l, o, w, e, r, n, s, t, i, d, </w>}
Step 1: most frequent pair = (l, o) → merge → "lo"
Step 2: most frequent pair = (lo, w) → merge → "low"
Step 3: most frequent pair = (e, s) → merge → "es"
Step 4: most frequent pair = (es, t) → merge → "est"
...continues until target vocab size reached
Notes: Walk through the merge steps slowly. The key insight is that common words end up as single tokens while rare words are decomposed into subword pieces.
Lesson 20 · Slide 4
Other Subword Algorithms
WordPiece (BERT)
Similar to BPE but selects merges that maximize the likelihood of the training data rather than raw frequency. Uses ## prefix for continuation subwords.
"tokenization" → ["token", "##ization"]
Unigram (SentencePiece)
Starts with a large vocabulary and prunes tokens that contribute least to the training data likelihood. Works top-down instead of bottom-up.
SentencePiece
A language-agnostic framework that treats input as a raw byte stream — no pre-tokenization needed. Supports both BPE and Unigram. Used by T5, Llama, Gemma.
Treats spaces as ▁ (Unicode block character) — no word boundary assumptions.
Byte-Level BPE
Used by GPT-2/3/4. Operates on UTF-8 bytes rather than Unicode characters. Guarantees any text can be tokenized. Base vocabulary is exactly 256 bytes.
Lesson 20 · Slide 5
Special Tokens
Every tokenizer reserves special tokens that carry structural meaning rather than text content. These tokens are critical for model behavior but are never generated from raw text.
| Token | Model Family | Purpose |
|---|
[CLS] | BERT | Classification token — its final hidden state represents the entire sequence |
[SEP] | BERT | Separator between sentence pairs (e.g., question and passage) |
[PAD] | Most models | Padding for batch processing — ignored via attention mask |
[MASK] | BERT | Masked token for pre-training (predict the hidden word) |
<|endoftext|> | GPT family | End of document / sequence boundary |
<s> / </s> | Llama, T5 | Beginning / end of sequence markers |
<|im_start|> | ChatGPT | Chat message boundary (role prefix) |
Notes: Emphasize that special tokens are added by the tokenizer — not typed by users. When counting tokens for cost estimation, remember that the API adds special tokens automatically.
Lesson 20 · Slide 6
Token Embeddings
After tokenization, each token ID is mapped to a dense vector via an embedding lookup table. This is where discrete symbols become continuous representations that the model can process.
The Embedding Pipeline
Text → Tokenizer → Token IDs [4821, 318, 257, 6827] → Embedding Table → Vectors [4096-dim each] → + Positional Encoding → Input to Transformer
Key Properties
- The embedding table has shape
(vocab_size, hidden_dim) — e.g., (100K, 4096) = 400M parameters just for embeddings
- Embeddings are learned during pre-training — similar tokens cluster together
- The same table is often tied to the output layer (weight tying), reducing total parameters
- Vocabulary size directly affects model size: doubling the vocabulary adds hundreds of millions of parameters
Notes: This connects back to Lesson 01 where we introduced embeddings briefly. Now students see exactly where they sit in the pipeline.
Lesson 20 · Slide 7
tiktoken vs SentencePiece & Multilingual Impact
The two dominant tokenizer implementations in production today have very different design philosophies.
tiktoken (OpenAI)
- Byte-level BPE
- Regex-based pre-splitting
- Rust backend, very fast
- Used by GPT-3.5/4, cl100k_base
- English-optimized vocabulary
SentencePiece (Google)
- BPE or Unigram
- No pre-tokenization required
- C++ backend
- Used by Llama, T5, Gemma
- Better multilingual coverage
Multilingual Token Efficiency
| Language | Text | cl100k_base tokens | Approx. ratio vs English |
|---|
| English | "Hello, how are you?" | 6 | 1.0x |
| Spanish | "Hola, como estas?" | 7 | ~1.2x |
| Japanese | "こんにちは、お元気ですか？" | 11 | ~1.8x |
| Hindi | "नमस्ते, आप कैसे हैं?" | 24 | ~4.0x |
| Thai | "สวัสดีครับ สบายดีไหม" | 22 | ~3.7x |
Cost implication: The same meaning in Hindi can cost 4x more than in English due to tokenization inefficiency.
Lesson 20 · Slide 8
Lab & Homework
Lab Exercises
- BPE from Scratch — implement the full BPE training algorithm in Python: character splitting, pair counting, merging, and vocabulary building
- Tokenizer Comparison — tokenize the same text with tiktoken and SentencePiece; compare token counts, vocabulary overlap, and subword boundaries
- Multilingual Analysis — measure token counts for equivalent sentences across 5+ languages; calculate cost multipliers relative to English
Homework Tasks
- Extend your BPE implementation with byte-level support and compare its vocabulary to tiktoken's cl100k_base on a sample corpus.
- Research how Llama 3's tokenizer improved multilingual efficiency over Llama 2. Write a 200-word summary of the changes and their impact.
- Design a cost-efficiency experiment: given a multilingual application, estimate the annual cost difference between an English-optimized tokenizer and a balanced multilingual tokenizer.