Lab Guide

Format: Python script (lab.py) · Time estimate: 45–60 minutes · Prerequisites: Python 3.10+, anthropic, tiktoken, python-dotenv installed

📥 Lab Files

⬇ lab.py (student) ⬇ lab_answer.py (solution)

Setup

Before starting, ensure your environment is ready:

# From the project root
pip install anthropic tiktoken python-dotenv

# Ensure your .env file has your API key
# ANTHROPIC_API_KEY=your_key_here

Run the lab script with python lesson24-when-to-finetune/lab.py. The student version has TODO markers where you need to fill in code.

Exercise 1: Use-Case Analyser

Goal

Apply the decision framework from the slides to evaluate real-world scenarios and recommend the right approach: prompt engineering, RAG, or fine-tuning.

Steps

Define a list of 6 use-case scenarios as dictionaries, each containing the task description, data availability, volume, and whether the problem is about knowledge or behaviour. The scenarios are:
- A: A legal firm needs to generate contract summaries that always follow a specific 5-section format with precise legal terminology. They have 2,000 approved examples. Volume: 500 contracts/day.
- B: A startup wants their chatbot to answer questions about their product documentation, which is updated weekly. They have 50 FAQ examples. Volume: 200 queries/day.
- C: A hospital needs radiology reports written in a standardised structure using ICD-10 codes. They have 10,000 de-identified reports. Volume: 300 reports/day.
- D: A developer wants to summarise Git commit messages into release notes. They have 20 example summaries. Volume: 5 per week.
- E: An e-commerce company wants product descriptions that match their playful brand voice. They have 800 approved descriptions. Volume: 1,000/day.
- F: A news agency needs to classify breaking news articles into 12 categories. Categories change quarterly. They have 3,000 labelled articles. Volume: 2,000/day.
For each scenario, implement a function evaluate_use_case(scenario) that walks through the 5-question decision framework and returns a recommendation with reasoning.
Print a formatted summary table showing each scenario, the recommendation, and the key factors that drove the decision.

Expected Output

=== Use-Case Analysis ===

Scenario A: Legal contract summaries
  Recommendation: FINE-TUNE
  Reasoning: Behaviour problem (format/terminology), 2000 examples available,
             high volume (500/day) justifies training cost.

Scenario B: Product documentation chatbot
  Recommendation: RAG
  Reasoning: Knowledge problem (weekly updates), insufficient data (50 examples),
             moderate volume. RAG handles changing content naturally.

Scenario C: Radiology reports
  Recommendation: FINE-TUNE
  Reasoning: Behaviour problem (structured format, domain terminology),
             10000 examples available, high volume.

Scenario D: Git commit summaries
  Recommendation: PROMPT ENGINEERING
  Reasoning: General task, tiny dataset (20 examples), very low volume (5/week).
             Not worth training investment.

Scenario E: Brand-voice product descriptions
  Recommendation: FINE-TUNE
  Reasoning: Behaviour problem (brand voice), 800 examples available,
             high volume (1000/day) ensures quick cost recovery.

Scenario F: News classification
  Recommendation: RAG + PROMPT ENGINEERING
  Reasoning: Categories change quarterly (knowledge shifts), classification is a
             general task. Use RAG for category definitions, prompting for task.

Discussion

Scenario F is deliberately ambiguous. Some students may argue for fine-tuning since 3,000 examples and high volume would justify it. The counter-argument is that quarterly category changes mean retraining every 3 months. Both answers are defensible — the important thing is articulating the trade-off.

Exercise 2: Prompt-vs-Fine-Tune Comparison

Goal

Test a style-consistency task with prompt engineering and measure where it falls short — demonstrating the gap that fine-tuning would fill.

Steps

Define a target style: terse, technical, no adjectives, always uses bullet points, British English spelling. Write a system prompt that encodes this style with 3 examples.
Create 10 test inputs — short requests for descriptions of various technical concepts (e.g., "Describe how a load balancer works", "Explain database indexing").
Call the Claude API with each test input using your system prompt.
Score each response on 4 criteria (each 0 or 1):
- Uses bullet points
- No adjectives (or minimal — fewer than 2)
- British English spelling (e.g., "optimise" not "optimize")
- Terse (under 100 words)
Calculate the overall compliance rate (score out of 40).

Expected Output

=== Style Compliance Test ===

Test 1: "Describe how a load balancer works"
  Bullets: YES | No adjectives: YES | British spelling: NO | Terse: YES
  Score: 3/4

Test 2: "Explain database indexing"
  Bullets: YES | No adjectives: NO | British spelling: YES | Terse: YES
  Score: 3/4

... (8 more tests)

=== Summary ===
Overall compliance: 28/40 (70%)
Bullet points:      9/10
No adjectives:      6/10
British spelling:   5/10
Terse:              8/10

Weakest criterion: British spelling (50%)
This is the type of consistent behavioural pattern that fine-tuning addresses.

Key Insight

You will likely find that the model follows some style rules consistently (like bullet points) but struggles with others (like always using British spelling). This inconsistency is exactly the problem fine-tuning solves — it shifts the model's default behaviour rather than relying on per-call instructions.

Exercise 3: Cost Calculator

Goal

Build a break-even calculator that compares the total cost of prompt engineering against fine-tuning over time, accounting for training costs and reduced per-call inference costs.

Steps

Define cost parameters for two approaches:

# Approach A: Prompt engineering with Claude Sonnet
prompt_eng = {
    "system_prompt_tokens": 2000,
    "avg_user_tokens": 300,
    "avg_output_tokens": 200,
    "input_price_per_m": 3.00,   # $/M tokens
    "output_price_per_m": 15.00,
}

# Approach B: Fine-tuned GPT-4o-mini (no system prompt needed)
fine_tuned = {
    "system_prompt_tokens": 0,
    "avg_user_tokens": 300,
    "avg_output_tokens": 200,
    "input_price_per_m": 0.30,   # $/M tokens (fine-tuned mini)
    "output_price_per_m": 1.20,
    "training_cost": 25.00,      # one-time
}

Write a function calculate_cost(config, num_calls) that returns total cost for a given number of API calls.
Write a function find_break_even(prompt_config, ft_config) that finds the number of calls at which fine-tuning becomes cheaper.
Print a comparison table showing costs at 1K, 10K, 100K, and 1M calls.
Convert the break-even call count to days, given a configurable calls_per_day parameter.

Expected Output

=== Cost Comparison ===

                   Prompt Eng.    Fine-Tuned    Savings
  1,000 calls:       $9.90          $25.33      -$15.43  (FT more expensive)
  10,000 calls:      $99.00         $28.30       $70.70  (FT cheaper)
  100,000 calls:     $990.00        $58.00      $932.00  (FT cheaper)
  1,000,000 calls:   $9,900.00     $355.00    $9,545.00  (FT cheaper)

Break-even point: ~2,830 calls

=== Break-Even Timeline ===
  At 100 calls/day:  ~28 days
  At 500 calls/day:  ~6 days
  At 1,000 calls/day: ~3 days
  At 10,000 calls/day: <1 day

Important Caveats

These numbers are illustrative. Real costs depend on your specific prompt lengths, output lengths, and provider pricing.
The calculator does not account for quality differences — a fine-tuned model may produce better or worse results than the prompted approach.
Retraining costs (when your data or requirements change) should be factored into the total cost of ownership.

Finishing Up

Once you have completed all three exercises:

Run the full script end-to-end: python lesson24-when-to-finetune/lab.py
Verify all exercises produce output without errors.
Compare your implementation with lab_answer.py — focus on the decision logic in Exercise 1 and the cost calculations in Exercise 3.

Self-Check

✓ Exercise 1: Can you justify each recommendation using the 5-question framework?
✓ Exercise 1: Can you identify which scenarios are ambiguous and argue both sides?
✓ Exercise 2: Did your compliance test reveal inconsistencies in style adherence?
✓ Exercise 2: Can you explain why some style rules are harder for the model to follow consistently?
✓ Exercise 3: Does your break-even analysis account for both training and inference costs?
✓ Exercise 3: Can you explain when fine-tuning does not make financial sense?