Lesson 33 · Slide 1
Custom Training
Training language models from scratch or via continued pre-training — when fine-tuning is not enough, and you need full control over what the model knows.
Learning Objectives
- Distinguish when custom training is needed versus fine-tuning or prompting
- Understand continued pre-training on domain-specific corpora
- Configure distributed training with DeepSpeed and FSDP
- Apply scaling laws (Chinchilla) to plan compute-optimal training runs
- Estimate cost and infrastructure requirements for training at scale
Notes: This is the final lesson in the course. It ties together everything from Module 5 (fine-tuning) and Module 7 (production) into the most advanced workflow: training your own model.
Lesson 33 · Slide 2
When Custom Training?
Most teams never need to train from scratch. Here is the decision hierarchy:
Prompting / RAG
- No training data required
- Works out of the box
- Best when: task is well-served by general knowledge + retrieval
Fine-Tuning (LoRA / Full)
- Adapts style, format, or narrow domain
- Hundreds to thousands of examples
- Best when: base model understands the domain but needs behavioural shaping
Continued Pre-Training
- Injects new domain knowledge (medical, legal, code)
- Millions of tokens of domain text
- Best when: the base model lacks core knowledge for your domain
Training from Scratch
- Full control over architecture, data, and tokeniser
- Billions of tokens, significant GPU budget
- Best when: no existing model fits your language, modality, or constraints
Lesson 33 · Slide 3
Continued Pre-Training
The most practical form of custom training: take an existing pre-trained model and continue training on your domain corpus. The model retains its general capabilities while absorbing new knowledge.
Key Decisions
- Base model: Llama 3, Mistral, or another open-weight model with a permissive licence
- Corpus size: Typically 1–50 B tokens of in-domain text (papers, code, docs, logs)
- Learning rate: Much lower than from-scratch training — typically 1e-5 to 5e-5
- Catastrophic forgetting: Mix in general-domain data (5–20%) to preserve broad capabilities
Example: Domain-Adapted Models
BloombergGPT — 50 B params trained on 363 B tokens of financial data mixed with general text. BioMistral — Mistral 7B continued pre-training on PubMed articles. CodeLlama — Llama 2 with continued pre-training on 500 B tokens of code.
Lesson 33 · Slide 4
Training Infrastructure
Custom training requires distributed computation across multiple GPUs. Two dominant frameworks handle the complexity:
DeepSpeed (Microsoft)
- ZeRO Stage 1: Partition optimiser states
- ZeRO Stage 2: + partition gradients
- ZeRO Stage 3: + partition parameters
- Offload to CPU/NVMe for memory savings
- Integrates with HuggingFace Trainer
FSDP (PyTorch Native)
- Fully Sharded Data Parallel
- Shards model parameters across GPUs
- All-gather before forward, reduce-scatter after backward
- No extra library — built into PyTorch 2.x
- Growing ecosystem support
Hardware Tiers
| Scale | Hardware | Typical Use |
|---|
| Hobby / Research | 1–4 x A100 80 GB | 7B model, continued pre-training |
| Startup | 8–64 x A100/H100 | 13–70 B model training |
| Frontier Lab | 1000+ H100s / TPU pods | 100 B+ from scratch |
Lesson 33 · Slide 5
Scaling Laws & Compute-Optimal Training
Scaling laws let you predict model performance before committing to a training run.
Chinchilla Scaling Law (Hoffmann et al., 2022)
For a compute-optimal model, the number of training tokens should scale linearly with the number of parameters. The optimal ratio is approximately 20 tokens per parameter.
| Model Size | Optimal Tokens | Approx. Compute (FLOPs) |
|---|
| 1 B | 20 B | ~1.2 x 1020 |
| 7 B | 140 B | ~5.9 x 1021 |
| 70 B | 1.4 T | ~5.9 x 1023 |
| 175 B | 3.5 T | ~3.7 x 1024 |
FLOPs Approximation
A rough estimate: FLOPs ≈ 6 × N × D, where N = parameters and D = training tokens. This 6x factor accounts for the forward and backward passes across all layers.
Notes: Llama 2 70B was trained on 2T tokens — slightly over-trained relative to Chinchilla, which is common because inference cost favours smaller, over-trained models.
Lesson 33 · Slide 6
Training Data Pipelines & Techniques
Data Pipeline at Scale
- Collection: Web scrapes (CommonCrawl), domain corpora, curated datasets
- Deduplication: MinHash / SimHash to remove near-duplicates (can remove 30–50% of web data)
- Quality filtering: Perplexity-based filtering, classifier-based quality scores
- Toxic content removal: Safety classifiers, PII scrubbing
- Tokenisation: Train a custom BPE tokeniser on your domain mix
- Packing: Concatenate documents to fill context-length sequences, minimising padding waste
Curriculum Learning
Present training data in a deliberate order rather than randomly. Start with cleaner, simpler data and progressively introduce harder, noisier, or more domain-specific examples. Research shows this can improve convergence speed by 10–20%.
Mixed-Precision Training
BF16 (Brain Float 16) is the standard for modern training. It maintains the same exponent range as FP32 while halving memory. Combined with FP32 master weights for the optimiser, this cuts memory usage nearly in half with negligible quality loss.
Lesson 33 · Slide 7
Checkpointing, Fault Tolerance & Cost
Checkpointing
- Save model weights + optimiser state + scheduler state + RNG state every N steps
- A 7 B model checkpoint is ~28 GB (FP32 weights + optimiser); ~56 GB with ZeRO Stage 3 aggregated
- Use async checkpointing to avoid stalling training while writing to disk
- Keep the last 3–5 checkpoints; delete older ones to manage storage
Fault Tolerance
At scale, hardware failures are not edge cases — they are expected. A 1000-GPU cluster will see node failures weekly. Strategies:
- Elastic training: frameworks like TorchElastic can restart on fewer nodes
- Redundant checkpointing: write to distributed storage (S3, GCS) not local disk
- Heartbeat monitoring: detect and replace failed nodes automatically
Cost Estimation
| Model | GPUs | Time | Cloud Cost (est.) |
|---|
| 7 B (140 B tokens) | 8 x A100 | ~7 days | ~$15,000 |
| 13 B (260 B tokens) | 32 x A100 | ~10 days | ~$80,000 |
| 70 B (1.4 T tokens) | 256 x H100 | ~30 days | ~$2,000,000 |
Lesson 33 · Slide 8
Future Directions & Lab
Emerging Trends
- Mixture of Experts (MoE): Only activate a subset of parameters per token — enables larger models at lower inference cost (e.g., Mixtral 8x7B)
- Multi-modal training: Joint training on text + images + audio + video from the start, not bolted on after
- Synthetic data: Using strong models to generate training data for smaller models (distillation at scale)
- Longer contexts via architecture: Ring Attention, Infini-Attention, and other approaches to push beyond 1M tokens efficiently
- Post-training recipes: RLHF, DPO, and their successors continue to evolve rapidly
Lab Preview
In today's lab you will:
- Set up a small training run with HuggingFace Trainer and a GPT-2 model
- Configure DeepSpeed ZeRO Stage 2 for memory-efficient training
- Analyse training curves (loss, learning rate, throughput)
- Estimate compute costs for scaling the run to larger models