Lab Guide

Format: Jupyter notebook (lab.ipynb) · Time estimate: 45–60 minutes · Prerequisites: Python 3.9+, tiktoken, sentencepiece installed

Setup

Before starting, ensure your environment is ready:

# From the project root
pip install tiktoken sentencepiece

No API keys are needed for this lab — everything runs locally.

Exercise 1: Implement BPE from Scratch

Goal

Build the Byte Pair Encoding algorithm from the ground up to understand how vocabularies are constructed.

Steps

Start with a small training corpus:

corpus = [
    "low low low low low",
    "lower lower",
    "newest newest newest",
    "widest widest",
]

Write a function get_vocab(corpus) that splits each word into characters and appends an end-of-word marker </w>:

def get_vocab(corpus):
    """Convert corpus into word-frequency dictionary with character-level splits.

    Returns:
        dict: e.g., {('l','o','w','</w>'): 5, ('l','o','w','e','r','</w>'): 2, ...}
    """
    # TODO: implement

Write a function get_pair_counts(vocab) that counts all adjacent token pairs across the vocabulary:

def get_pair_counts(vocab):
    """Count frequency of each adjacent pair in the vocabulary.

    Returns:
        dict: e.g., {('l','o'): 7, ('o','w'): 7, ('w','</w>'): 5, ...}
    """
    # TODO: implement

Write a function merge_pair(pair, vocab) that merges every occurrence of the most frequent pair:

def merge_pair(pair, vocab):
    """Replace all occurrences of `pair` in vocab with merged token.

    Returns:
        dict: updated vocabulary with the pair merged into a single token
    """
    # TODO: implement

Write the main training loop that performs num_merges iterations:

def train_bpe(corpus, num_merges=10):
    """Train BPE on corpus for num_merges iterations.

    Returns:
        list: ordered list of merge operations [(pair, new_token), ...]
        dict: final vocabulary
    """
    vocab = get_vocab(corpus)
    merges = []

    for i in range(num_merges):
        pairs = get_pair_counts(vocab)
        if not pairs:
            break
        best_pair = max(pairs, key=pairs.get)
        vocab = merge_pair(best_pair, vocab)
        new_token = best_pair[0] + best_pair[1]
        merges.append((best_pair, new_token))
        print(f"Merge {i+1}: {best_pair} -> '{new_token}' (freq: {pairs[best_pair]})")

    return merges, vocab

Run the training and inspect the merge log. Verify that common words like "low" get merged into single tokens early.

Expected Output

Merge 1: ('e', 's') -> 'es' (freq: 5)
Merge 2: ('es', 't') -> 'est' (freq: 5)
Merge 3: ('l', 'o') -> 'lo' (freq: 7)
Merge 4: ('lo', 'w') -> 'low' (freq: 7)
Merge 5: ('n', 'e') -> 'ne' (freq: 3)
Merge 6: ('ne', 'w') -> 'new' (freq: 3)
Merge 7: ('new', 'est') -> 'newest' (freq: 3)
Merge 8: ('low', '</w>') -> 'low</w>' (freq: 5)
Merge 9: ('w', 'i') -> 'wi' (freq: 2)
Merge 10: ('wi', 'd') -> 'wid' (freq: 2)

Key Insight

Notice how the algorithm naturally discovers meaningful subwords. "est" emerges as a reusable suffix, and "low" becomes a complete token because it appears frequently. This is exactly how production tokenizers build their vocabularies — just at a much larger scale (trained on billions of words with 50K–100K merges).

Exercise 2: Compare Tokenizers

Goal

Understand the practical differences between tiktoken (byte-level BPE) and SentencePiece by tokenizing the same text with both.

Steps

Install and load both tokenizers:

import tiktoken
import sentencepiece as spm

# tiktoken: cl100k_base (GPT-4 family)
enc = tiktoken.get_encoding("cl100k_base")

# SentencePiece: download a pre-trained model or train a small one
# For this exercise, we'll train a small model on sample text
with open("sample_corpus.txt", "w") as f:
    f.write("This is a sample corpus for training a small SentencePiece model. " * 100)
    f.write("Tokenization is the process of converting text into tokens. " * 100)
    f.write("Machine learning models process numerical representations. " * 100)

spm.SentencePieceTrainer.Train(
    input="sample_corpus.txt",
    model_prefix="sp_model",
    vocab_size=500,
    model_type="bpe"
)
sp = spm.SentencePieceProcessor()
sp.Load("sp_model.model")

Tokenize a set of test strings with both:

test_strings = [
    "Hello, world!",
    "The quick brown fox jumps over the lazy dog.",
    "def fibonacci(n): return n if n < 2 else fibonacci(n-1) + fibonacci(n-2)",
    "Tokenization is fundamentally important.",
    "supercalifragilisticexpialidocious",
]

for s in test_strings:
    tk_tokens = enc.encode(s)
    sp_tokens = sp.Encode(s, out_type=str)
    print(f"\nText: {s}")
    print(f"  tiktoken ({len(tk_tokens)} tokens): {[enc.decode([t]) for t in tk_tokens]}")
    print(f"  SentencePiece ({len(sp_tokens)} tokens): {sp_tokens}")

Compare:
- Which tokenizer produces fewer tokens for each string?
- How do they handle spaces differently?
- What happens with the rare word "supercalifragilisticexpialidocious"?

Key Observations to Note

tiktoken uses regex-based pre-splitting (whitespace, punctuation) before BPE. SentencePiece treats the raw stream as-is.
SentencePiece represents spaces as ▁ (a visible character). tiktoken handles spaces as part of byte sequences.
With a small vocabulary (500), the SentencePiece model will produce more tokens than tiktoken (100K vocabulary). In production, vocabulary sizes are comparable.

Exercise 3: Multilingual Token Analysis

Goal

Measure the token efficiency gap across languages and calculate its cost impact.

Steps

Define equivalent sentences in multiple languages:

sentences = {
    "English":    "The weather is beautiful today and I am going for a walk in the park.",
    "Spanish":    "El clima es hermoso hoy y voy a dar un paseo por el parque.",
    "French":     "Le temps est magnifique aujourd'hui et je vais me promener dans le parc.",
    "German":     "Das Wetter ist heute wunderschoen und ich gehe im Park spazieren.",
    "Russian":    "Сегодня прекрасная погода, и я иду гулять в парке.",
    "Japanese":   "今日はとても良い天気なので、公園を散歩します。",
    "Chinese":    "今天天气很好，我要去公园散步。",
    "Hindi":      "आज मौसम बहुत अच्छा है और मैं पार्क में टहलने जा रहा हूँ।",
    "Arabic":     "الطقس جميل اليوم وسأذهب للتنزه في الحديقة.",
    "Korean":     "오늘 날씨가 아름다워서 공원에 산책하러 갑니다.",
}

Tokenize each sentence and calculate statistics:

enc = tiktoken.get_encoding("cl100k_base")

print(f"{'Language':<12} {'Chars':>6} {'Tokens':>7} {'Chars/Tok':>10} {'vs English':>11}")
print("-" * 50)

en_tokens = len(enc.encode(sentences["English"]))

for lang, text in sentences.items():
    tokens = enc.encode(text)
    n_chars = len(text)
    n_tokens = len(tokens)
    ratio = n_chars / n_tokens
    vs_en = n_tokens / en_tokens
    print(f"{lang:<12} {n_chars:>6} {n_tokens:>7} {ratio:>10.1f} {vs_en:>10.1f}x")

Calculate the annual cost difference for a multilingual application:

# Assume: 10,000 API calls/day, average 500 tokens/call (English baseline)
# Model: claude-3-5-sonnet at $3/M input tokens

calls_per_day = 10_000
base_tokens = 500
price_per_m = 3.00
days_per_year = 365

for lang, text in sentences.items():
    tokens = len(enc.encode(text))
    multiplier = tokens / en_tokens
    adjusted_tokens = base_tokens * multiplier
    annual_cost = (adjusted_tokens * calls_per_day * days_per_year / 1_000_000) * price_per_m
    print(f"{lang:<12}: {multiplier:.1f}x tokens -> ${annual_cost:,.0f}/year")

Expected Output

Language     Chars  Tokens  Chars/Tok   vs English
--------------------------------------------------
English         70      16        4.4        1.0x
Spanish         62      16        3.9        1.0x
French          73      19        3.8        1.2x
German          63      16        3.9        1.0x
Russian         52      20        2.6        1.3x
Japanese        20      14        1.4        0.9x
Chinese         15      12        1.2        0.8x
Hindi           56      38        1.5        2.4x
Arabic          44      24        1.8        1.5x
Korean          24      18        1.3        1.1x

Important Caveat

Exact token counts vary with the specific text and tokenizer version. The ratios above are approximate. Run the code yourself to get precise numbers for the current cl100k_base encoding. The key takeaway is the relative difference, not the absolute counts.

Discussion Questions

Why do Chinese and Japanese sometimes produce fewer tokens than English, despite using non-Latin characters?
Why does Hindi consistently require more tokens than other languages?
If you were building a customer support chatbot serving users in 10 languages, how would tokenization efficiency affect your architecture decisions?

Finishing Up

Once you have completed all three exercises:

Run all cells from top to bottom to verify everything executes without errors (Kernel → Restart & Run All).
Save the notebook.
Compare your BPE implementation with the reference solution — focus on edge cases and efficiency.

Self-Check

✓ Exercise 1: Your BPE implementation correctly identifies and merges the most frequent pairs in order.
✓ Exercise 1: You can explain why "low" becomes a single token before "lower" does.
✓ Exercise 2: You can describe three differences between tiktoken and SentencePiece tokenization.
✓ Exercise 2: You understand why SentencePiece uses the ▁ character for spaces.
✓ Exercise 3: You can explain why Hindi uses more tokens than English for the same meaning.
✓ Exercise 3: You can calculate the cost multiplier for a non-English language given token counts.