Homework

Task 1: Training Cost Analysis

Goal: Develop the ability to estimate training costs and make informed build-vs-buy decisions.

Instructions

Your company is considering training a domain-specific language model for internal use. You have been asked to prepare a cost comparison. Using the Chinchilla scaling law and the FLOPs approximation formula (FLOPs = 6 x N x D), fill in the following table for three model sizes:

Model Size	Optimal Tokens (Chinchilla)	Total FLOPs	A100 GPU Hours (at 40% MFU)	Estimated Cloud Cost
1.3 B	?	?	?	?
7 B	?	?	?	?
13 B	?	?	?	?

Assumptions to use:

• Chinchilla optimal ratio: 20 tokens per parameter
• A100 80GB BF16 peak throughput: 312 TFLOPS
• Model FLOPs Utilisation (MFU): 40%
• A100 on-demand cloud cost: $2.00 per GPU-hour

Then answer these questions:

1. Your domain corpus contains 50 B tokens. Which model size is the best fit, and why?
2. If you used 8 GPUs, how many wall-clock days would each training run take?
3. Compare the training cost of the 7 B model against 12 months of Claude API usage at $10,000/month. At what monthly API spend does training your own model break even (ignoring inference costs)?

Self-check

• ✓ Your FLOPs calculations are in the right order of magnitude (10^20 for 1.3B, 10^22 for 7B, 10^22 for 13B).
• ✓ GPU hours account for the 40% utilisation factor (i.e., you divided by 0.4).
• ✓ Your cost estimates are within 2x of published training cost reports for similar models.
• ✓ Your break-even analysis considers that a trained model also has ongoing inference costs.

Task 2: Data Pipeline Design

Goal: Design a training data pipeline for a realistic domain.

Instructions

Choose one of the following domains (or pick your own):

• Legal: Court opinions, contracts, statutes, legal commentary
• Biomedical: PubMed abstracts, clinical notes (synthetic), drug labels
• Financial: SEC filings, earnings transcripts, financial news

Write a 300–500 word design document that covers:

1. Data sources: Where would you obtain training data? List 3–5 specific sources with approximate token counts.
2. Deduplication strategy: How would you handle duplicates? What tool or algorithm would you use?
3. Quality filtering: What criteria define "high quality" for your domain? How would you implement filtering?
4. Safety and compliance: What PII or sensitive data concerns exist? How would you address them?
5. Data mixing: What percentage of general-domain data would you include, and why?
6. Tokeniser decision: Would you train a custom tokeniser or reuse an existing one? Justify your choice.

Self-check

• ✓ You have identified at least 3 concrete data sources with estimated sizes.
• ✓ Your deduplication strategy names a specific algorithm (MinHash, SimHash, exact dedup, etc.).
• ✓ Your quality filtering criteria are domain-specific, not generic.
• ✓ You have considered PII and licensing constraints specific to your chosen domain.
• ✓ Your data mixing ratio is justified with a reason, not just a number.

Task 3: Scaling Laws Exploration

Goal: Build intuition for how scaling laws guide training decisions.

Instructions

Write a Python script that plots the Chinchilla-optimal training configuration across a range of compute budgets. Your script should:

1. Define a range of compute budgets from 10^18 to 10^24 FLOPs.
2. For each budget, compute the optimal model size (N) and optimal number of training tokens (D) using the Chinchilla relationship: D = 20 * N and FLOPs = 6 * N * D. Solve for N: N = sqrt(FLOPs / 120).
3. Plot two charts:
   • Compute budget (x-axis, log scale) vs. optimal model size in billions of parameters (y-axis, log scale)
   • Compute budget (x-axis, log scale) vs. optimal training tokens in billions (y-axis, log scale)
4. Annotate the plot with known models: GPT-2 (1.5B, 40B tokens), Llama 2 7B (7B, 2T tokens), Chinchilla (70B, 1.4T tokens). Mark whether each model was over-trained, under-trained, or near-optimal relative to its compute budget.

Self-check

• ✓ Your plots use log-log scales and show a clear linear relationship.
• ✓ GPT-3 / GPT-2 appear as under-trained (trained on fewer tokens than Chinchilla recommends for their size).
• ✓ Llama 2 7B appears as over-trained (trained on far more tokens than the 140B Chinchilla recommends).
• ✓ You can explain in one sentence why over-training is a rational choice despite being compute-suboptimal.