Homework

Task 1: Byte-Level BPE Extension

Goal: Extend your BPE implementation from the lab to operate on bytes instead of characters, matching the approach used by GPT-2/3/4 and tiktoken.

Instructions

1. Take the BPE implementation you wrote in Exercise 1 of the lab.
2. Modify it to start from UTF-8 byte sequences instead of Unicode characters:

   # Instead of splitting "Hello" into ['H','e','l','l','o']
   # Split into bytes: [72, 101, 108, 108, 111]
   text = "Hello"
   byte_seq = list(text.encode("utf-8"))
   print(byte_seq)  # [72, 101, 108, 108, 111]

3. Train your byte-level BPE on a longer corpus — use at least 1000 words of English text (you can use any text file or a paragraph from Wikipedia).
4. Run 200 merges and examine the resulting vocabulary.
5. Compare the tokens your implementation produces with tiktoken's cl100k_base on 5 test strings. They will not match exactly (tiktoken was trained on a much larger corpus with 100K merges), but look for patterns:
   • Do both discover common English words as tokens?
   • How do they differ on rare words?
   • What about punctuation and whitespace?

Deliverable

A notebook or script that:

• Implements byte-level BPE training
• Prints the top 20 most common merged tokens (decoded back to strings)
• Shows a side-by-side comparison of your tokenizer vs tiktoken on 5 test strings

Self-check

• ✓ Your tokenizer can handle any UTF-8 input without errors (including non-English text and emoji).
• ✓ The base vocabulary is exactly 256 entries (one per byte value).
• ✓ You can explain why byte-level BPE guarantees no out-of-vocabulary tokens.
• ✓ Your comparison table shows at least 5 test strings with token counts from both tokenizers.

Task 2: Llama 3 Tokenizer Research

Goal: Understand how tokenizer design has evolved to address multilingual efficiency.

Instructions

Research the changes between Llama 2's tokenizer (32K vocabulary, SentencePiece BPE) and Llama 3's tokenizer (128K vocabulary, tiktoken-based). Write a 200-word summary addressing:

1. Vocabulary size: Why did Meta increase from 32K to 128K tokens? What are the trade-offs of a larger vocabulary?
2. Training data: How did the composition of the tokenizer training data change, and why does this matter for multilingual performance?
3. Measured impact: What concrete improvements were reported for non-English languages? (Look for specific numbers — e.g., "X% fewer tokens for language Y.")
4. Architecture impact: A larger vocabulary means a larger embedding table. How does this affect total model size and inference speed?

Resources to Start With

• The Llama 3 announcement blog post
• The Llama 3 technical report
• Hugging Face model cards for Llama 2 and Llama 3

Self-check

• ✓ Your summary is 180–220 words.
• ✓ You cite at least one specific number or statistic about the improvement.
• ✓ You explain both the benefits and costs of a larger vocabulary.
• ✓ A reader who has not read the Llama 3 paper would learn something concrete from your summary.

Task 3: Multilingual Cost Analysis

Goal: Apply tokenization knowledge to a realistic business scenario.

Instructions

You are designing a multilingual customer support chatbot that will serve users in English, Spanish, Japanese, Hindi, and Arabic. The expected usage is:

Language	% of traffic	Avg. conversation length (words)
English	40%	300
Spanish	25%	280
Japanese	15%	200
Hindi	12%	250
Arabic	8%	260

Total conversations per day: 50,000. Model: Claude Sonnet at $3/M input tokens, $15/M output tokens. Assume each conversation involves 4 API calls (2 user messages + 2 assistant responses), with output averaging 60% of input token length.

1. Using your token ratios from Exercise 3 of the lab, calculate the weighted average token multiplier across your user base.
2. Calculate the annual API cost broken down by language.
3. Calculate how much you would save if a hypothetical "perfectly balanced" tokenizer reduced all non-English languages to 1.0x English efficiency.
4. Write a brief recommendation (100–150 words) for the engineering team on whether tokenizer efficiency should influence model selection for this project.

Deliverable

A spreadsheet, notebook, or script that produces:

• A cost breakdown table by language (annual)
• The total annual cost
• The potential savings from a balanced tokenizer
• Your written recommendation

Self-check

• ✓ Your cost calculations account for both input and output tokens.
• ✓ You used measured token ratios (from the lab), not guesses.
• ✓ Your recommendation addresses the trade-off between tokenizer efficiency and other model selection criteria (quality, latency, features).
• ✓ The total annual cost is a reasonable number (tens of thousands of dollars, not millions or pennies).