Lesson 24 · Slide 1
When to Fine-Tune
A decision framework for choosing between prompting, RAG, and fine-tuning — so you reach for the right tool at the right time.
Learning Objectives
- Apply a structured decision framework to determine when fine-tuning is the right approach
- Identify the scenarios where fine-tuning outperforms prompt engineering and RAG
- Recognise anti-patterns — situations where fine-tuning wastes time and money
- Compare fine-tuning approaches: full, LoRA, QLoRA, API-based
- Estimate data requirements and costs for a fine-tuning project
Notes: This is the gateway lesson for Module 5. Students should leave with a clear mental model of when fine-tuning is justified versus when simpler approaches suffice.
Lesson 24 · Slide 2
The Three-Tier Approach
Before fine-tuning, always ask: can I solve this with a simpler method? Think of these as escalation tiers — start at the top and only move down when the simpler approach fails.
Tier 1: Prompt Engineering
- Zero cost beyond API calls
- Instant iteration — change the prompt, re-run
- Works for most general tasks
- No training data needed
Tier 2: RAG (Retrieval-Augmented Generation)
- Grounds responses in your data
- Knowledge can be updated without retraining
- Good for factual, document-heavy tasks
- Requires a vector store and retrieval pipeline
Tier 3: Fine-Tuning
- Changes the model's behaviour, not just its context
- Requires curated training data (hundreds to thousands of examples)
- Training cost + ongoing serving cost
- Best for style, format, and domain specialisation
Decision Rule
Try Tier 1 first. If the model knows the right answer but doesn't produce it in the right way, fine-tuning may help. If it lacks knowledge, try RAG first.
Lesson 24 · Slide 3
When Fine-Tuning Helps
Fine-tuning shines when you need to change how the model responds, not what it knows.
Strong Use Cases
- Style and tone consistency: matching a brand voice, writing in a specific persona, or adopting a house style across thousands of outputs
- Domain-specific terminology: medical coding (ICD-10), legal citation formats, financial report structures — where the model must use precise jargon correctly every time
- Structured output reliability: consistently producing valid JSON, XML, or custom schemas without format drift over long outputs
- Latency reduction: a fine-tuned smaller model can replace a larger model with long system prompts, cutting both latency and cost
- Cost optimisation at scale: replacing a 4000-token system prompt with learned behaviour saves millions of input tokens per month
The Key Signal
If you find yourself writing ever-longer system prompts with dozens of examples and the model still drifts from your desired format or tone, that is the strongest signal that fine-tuning will help.
Lesson 24 · Slide 4
When NOT to Fine-Tune
Fine-tuning is expensive and slow to iterate. Avoid it when simpler solutions work.
Small Datasets (< 100 examples)
Fine-tuning on too little data leads to overfitting. Few-shot prompting is usually better with small example sets.
Rapidly Changing Knowledge
If facts change weekly (stock prices, product inventory, news), fine-tuning bakes in stale data. Use RAG instead — update the index, not the model.
General-Purpose Tasks
Summarisation, translation, basic Q&A — base models already excel at these. Fine-tuning adds cost without meaningful improvement.
When You Haven't Tried Prompting
Teams often jump to fine-tuning before exhausting prompt engineering. A well-structured system prompt with examples solves most problems.
Notes: Ask students: "Has anyone been tempted to fine-tune when a better prompt would have sufficed?" This is extremely common in industry.
Lesson 24 · Slide 5
Decision Framework Flowchart
Walk through these questions in order to reach the right approach for your use case.
- Does the base model understand the task? Test with a clear prompt and a few examples. If yes, proceed to question 2. If no — the task may be too specialised; collect data and consider fine-tuning.
- Is the model's output format/style acceptable? If yes — use prompt engineering (Tier 1). If no, proceed to question 3.
- Does the model need external knowledge? If yes — implement RAG (Tier 2). If it already "knows" the answer but produces it incorrectly, proceed to question 4.
- Do you have 500+ high-quality labelled examples? If yes — fine-tuning is likely justified. If no — collect more data or improve your prompts.
- Is this a high-volume production task? If yes — fine-tuning a smaller model can reduce per-call cost by 5-20x. If no — the training investment may not pay off.
Notes: This flowchart is the core artifact of the lesson. Encourage students to print or bookmark it.
Lesson 24 · Slide 6
Fine-Tuning Approaches Overview
Not all fine-tuning is the same. The approach you choose depends on your resources, data, and deployment target.
| Approach | What Changes | GPU Needed | Data Size | Best For |
|---|
| Full Fine-Tuning | All model weights | 8+ A100s (80 GB) | 10K–100K+ examples | Maximum performance, large orgs |
| LoRA | Low-rank adapter matrices (~1-5% of params) | 1 A100 or 2 A10s | 500–10K examples | Best balance of quality and cost |
| QLoRA | LoRA on quantised (4-bit) base | 1 T4/L4 (16 GB) | 500–10K examples | Consumer/Colab hardware |
| API-Based | Provider handles training | None (cloud) | 100–10K examples | No ML infrastructure needed |
Recommendation for Most Teams
Start with API-based fine-tuning (OpenAI, Together AI, Fireworks) to validate that fine-tuning helps at all. Then graduate to LoRA/QLoRA on open-source models if you need more control, lower per-call cost, or data privacy.
Lesson 24 · Slide 7
Data Requirements & Cost Analysis
Fine-tuning is only as good as your training data. Here is what to budget for.
Data Requirements
Minimum Viable Dataset
- API fine-tuning: 100–500 examples
- LoRA (open-source): 500–2,000 examples
- Full fine-tuning: 10,000+ examples
Each example = one (input, output) pair that demonstrates the behaviour you want.
Data Quality Checklist
- Diverse — covers edge cases, not just the easy path
- Correct — human-verified outputs
- Consistent — same format and style across examples
- Representative — matches production distribution
Cost Comparison (Illustrative)
| Approach | Training Cost | Per-Call Inference | Break-Even at 10K calls/day |
|---|
| Prompt engineering (Sonnet) | $0 | ~$0.018 | N/A (baseline) |
| API fine-tuned (GPT-4o-mini) | ~$5–50 | ~$0.002 | ~1 day |
| LoRA on Llama 3 8B (self-hosted) | ~$10–50 (GPU hours) | ~$0.0005 | ~1 day |
| Full fine-tune Llama 3 70B | ~$500–2,000 | ~$0.003 | ~2 weeks |
Notes: These costs are approximate and vary by provider. The key insight is that fine-tuning has an upfront cost that pays off at scale — similar to buying vs renting.
Lesson 24 · Slide 8
Lab & Homework
Lab Exercises
- Use-Case Analyser — evaluate 6 real-world scenarios using the decision framework and recommend prompt engineering, RAG, or fine-tuning for each
- Prompt-vs-Fine-Tune Comparison — test a style-consistency task with prompt engineering, then estimate the improvement fine-tuning would provide
- Cost Calculator — build a break-even calculator comparing prompt engineering costs against fine-tuning costs at various call volumes
Homework Tasks
- Identify a real task from your work or a personal project. Walk through the 5-question decision framework and write up your recommendation with justification.
- Prepare a fine-tuning proposal: describe the task, estimate data requirements, calculate training and inference costs, and determine the break-even point.
- Research one API-based fine-tuning provider and summarise their data format requirements, pricing, and limitations.