Lab Guide
Format: Jupyter notebook (lab.ipynb) · Time estimate: 45–60 minutes · Prerequisites: Python 3.9+, anthropic, openai, google-generativeai, tiktoken, python-dotenv installed
Setup
Before starting, ensure your environment has API keys for all three providers:
# From the project root
pip install anthropic openai google-generativeai tiktoken python-dotenv
# Your .env file should contain:
# ANTHROPIC_API_KEY=your_key_here
# OPENAI_API_KEY=your_key_here
# GOOGLE_API_KEY=your_key_hereAPI Key Note
You need active API keys for at least two of the three providers to complete this lab. If you only have one, you can still complete Exercise 1 (matrix) and Exercise 3 (cost calculator) fully, and do a partial comparison in Exercise 2. Free tiers are available for all three providers.
Exercise 1: Model Comparison Matrix
Goal
Build a structured comparison matrix that evaluates 5+ models across the six selection criteria for a specific use case.
Steps
- Choose a concrete use case. Good examples:
- Customer support ticket classifier and responder
- Code review assistant for a development team
- Document summarisation pipeline for legal contracts
- Define the model pricing data structure:
MODELS = { "claude-3-5-haiku-20241022": { "provider": "Anthropic", "input_cost_per_m": 0.80, "output_cost_per_m": 4.00, "context_window": 200_000, "multimodal": True, "latency_tier": "fast", "self_hostable": False, }, "claude-sonnet-4-20250514": { "provider": "Anthropic", "input_cost_per_m": 3.00, "output_cost_per_m": 15.00, "context_window": 200_000, "multimodal": True, "latency_tier": "medium", "self_hostable": False, }, "gpt-4o": { "provider": "OpenAI", "input_cost_per_m": 2.50, "output_cost_per_m": 10.00, "context_window": 128_000, "multimodal": True, "latency_tier": "medium", "self_hostable": False, }, "gpt-4o-mini": { "provider": "OpenAI", "input_cost_per_m": 0.15, "output_cost_per_m": 0.60, "context_window": 128_000, "multimodal": True, "latency_tier": "fast", "self_hostable": False, }, "gemini-1.5-pro": { "provider": "Google", "input_cost_per_m": 1.25, "output_cost_per_m": 5.00, "context_window": 1_000_000, "multimodal": True, "latency_tier": "medium", "self_hostable": False, }, "gemini-1.5-flash": { "provider": "Google", "input_cost_per_m": 0.075, "output_cost_per_m": 0.30, "context_window": 1_000_000, "multimodal": True, "latency_tier": "fast", "self_hostable": False, }, "llama-3-70b": { "provider": "Meta (self-hosted)", "input_cost_per_m": 0.00, "output_cost_per_m": 0.00, "context_window": 8_192, "multimodal": False, "latency_tier": "variable", "self_hostable": True, }, } - Write a function that scores each model (1–5) on the six criteria for your chosen use case:
def score_model(model_name: str, model_info: dict, use_case: dict) -> dict: """Score a model 1-5 on each criterion for the given use case.""" scores = {} # Task complexity: does the model have enough capability? # (Use benchmark data or your own testing) scores["task_complexity"] = ... # TODO # Latency: does the model meet response time requirements? scores["latency"] = ... # TODO # Cost: how expensive per task at the expected volume? scores["cost"] = ... # TODO # Privacy: does deployment model meet compliance needs? scores["privacy"] = ... # TODO # Multimodal: does it support required modalities? scores["multimodal"] = ... # TODO # Context window: is it large enough for the task? scores["context_window"] = ... # TODO return scores - Print the comparison matrix as a formatted table.
Expected Output
=== Model Comparison Matrix ===
Use case: Customer support ticket classifier (50K tickets/day)
Model | Complexity | Latency | Cost | Privacy | Multimodal | Context | TOTAL
-----------------------|------------|---------|------|---------|------------|---------|------
claude-3-5-haiku | 4 | 5 | 4 | 3 | 5 | 5 | 26
claude-sonnet-4 | 5 | 3 | 2 | 3 | 5 | 5 | 23
gpt-4o | 5 | 3 | 3 | 3 | 5 | 4 | 23
gpt-4o-mini | 3 | 5 | 5 | 3 | 5 | 4 | 25
gemini-1.5-flash | 3 | 5 | 5 | 3 | 5 | 5 | 26
llama-3-70b | 4 | 4 | 5 | 5 | 1 | 2 | 21Discussion
The raw total score is a starting point, not a final answer. In practice, you would weight the criteria differently based on your priorities. A healthcare company might weight privacy at 3x, making llama-3-70b the clear winner despite its lower total score.
Exercise 2: Cross-Provider Testing
Goal
Send identical prompts to multiple providers and compare quality, latency, and cost empirically.
Steps
- Define a test suite of 5 prompts that represent your use case. Include a mix of difficulties:
TEST_PROMPTS = [ { "name": "simple_classification", "prompt": "Classify this support ticket as billing, technical, or general: " "'I can't log into my account after changing my password yesterday.'", "expected": "technical", }, { "name": "extraction", "prompt": "Extract the product name, issue type, and urgency (low/medium/high) " "as JSON from: 'Our enterprise dashboard has been down for 2 hours " "and 500 users are affected. This is critical.'", "expected_keys": ["product", "issue_type", "urgency"], }, { "name": "reasoning", "prompt": "A customer says: 'I was charged $49.99 but my plan is $29.99/month " "and I have a 20% annual discount.' Calculate what they should have " "been charged and explain the discrepancy.", "requires_math": True, }, { "name": "generation", "prompt": "Write a professional but empathetic response to a customer who " "received a damaged product and wants a replacement.", "quality_check": "tone", }, { "name": "long_context", "prompt": "Summarise the key action items from this meeting transcript: " + "[Insert a 2000+ word meeting transcript here]", "tests_context": True, }, ] - Write a unified test runner that calls each provider and records results:
import time import anthropic import openai import google.generativeai as genai async def test_model(provider: str, model: str, prompt: str) -> dict: """Call a model and record response, latency, and token usage.""" start = time.time() if provider == "anthropic": client = anthropic.Anthropic() resp = client.messages.create( model=model, max_tokens=500, messages=[{"role": "user", "content": prompt}] ) text = resp.content[0].text input_tokens = resp.usage.input_tokens output_tokens = resp.usage.output_tokens elif provider == "openai": client = openai.OpenAI() resp = client.chat.completions.create( model=model, max_tokens=500, messages=[{"role": "user", "content": prompt}] ) text = resp.choices[0].message.content input_tokens = resp.usage.prompt_tokens output_tokens = resp.usage.completion_tokens elif provider == "google": model_obj = genai.GenerativeModel(model) resp = model_obj.generate_content(prompt) text = resp.text input_tokens = resp.usage_metadata.prompt_token_count output_tokens = resp.usage_metadata.candidates_token_count elapsed = time.time() - start return { "model": model, "response": text, "input_tokens": input_tokens, "output_tokens": output_tokens, "latency_s": round(elapsed, 2), } - Run all prompts against at least 3 models (one per provider).
- For each prompt, manually rate the quality of each response on a 1–5 scale.
- Compute cost per prompt using the pricing data from Exercise 1.
- Print a results table for each prompt showing model, quality, latency, and cost.
Expected Output
=== Test: simple_classification ===
Model | Quality | Latency | Input Tok | Output Tok | Cost
-----------------------|---------|---------|-----------|------------|--------
claude-3-5-haiku | 5/5 | 0.34s | 42 | 3 | $0.000046
gpt-4o-mini | 5/5 | 0.28s | 39 | 2 | $0.000007
gemini-1.5-flash | 5/5 | 0.22s | 41 | 3 | $0.000004
=== Test: reasoning ===
Model | Quality | Latency | Input Tok | Output Tok | Cost
-----------------------|---------|---------|-----------|------------|--------
claude-3-5-haiku | 3/5 | 0.89s | 58 | 156 | $0.000670
gpt-4o-mini | 2/5 | 0.67s | 55 | 142 | $0.000094
gemini-1.5-flash | 3/5 | 0.51s | 57 | 138 | $0.000046Key Observations to Note
- Simple tasks often show identical quality across all models — the cheapest model wins.
- Complex reasoning tasks reveal the largest quality gaps between model tiers.
- Latency differences are most noticeable in long-generation tasks.
- Cost differences compound dramatically at scale: a $0.0001 difference per call is $3,000/month at 1M calls/day.
Exercise 3: Cost-Per-Task Calculator
Goal
Build a reusable cost calculator that compares total cost of ownership across models at different volume levels.
Steps
- Using the token usage data from Exercise 2 (or estimates), calculate cost per task for each model:
def cost_per_task(model_info: dict, avg_input_tokens: int, avg_output_tokens: int) -> float: """Calculate cost of a single task in USD.""" input_cost = (avg_input_tokens / 1_000_000) * model_info["input_cost_per_m"] output_cost = (avg_output_tokens / 1_000_000) * model_info["output_cost_per_m"] return input_cost + output_cost - Scale to monthly projections at different volumes:
VOLUME_TIERS = [100, 1_000, 10_000, 100_000, 1_000_000] # calls per day def monthly_cost(cost_per_call: float, calls_per_day: int) -> float: return cost_per_call * calls_per_day * 30 - Print a cost comparison table across all models and volume tiers.
- Identify the crossover points: at what volume does self-hosting become cheaper than API calls?
Expected Output
=== Monthly Cost Comparison (avg 200 input + 100 output tokens/task) ===
Model | 100/day | 1K/day | 10K/day | 100K/day | 1M/day
-----------------------|----------|----------|-----------|------------|----------
claude-3-5-haiku | $0.53 | $5.28 | $52.80 | $528.00 | $5,280
claude-sonnet-4 | $2.25 | $22.50 | $225.00 | $2,250.00 | $22,500
gpt-4o | $1.65 | $16.50 | $165.00 | $1,650.00 | $16,500
gpt-4o-mini | $0.09 | $0.90 | $9.00 | $90.00 | $900
gemini-1.5-flash | $0.05 | $0.47 | $4.65 | $46.50 | $465
llama-3-70b (hosted) | $2,160* | $2,160* | $2,160* | $2,160* | $2,160*
* Self-hosted cost is fixed GPU rental (~$3/hr for A100 = $2,160/month)
Breakeven vs Haiku API: ~12,300 calls/day
Breakeven vs Sonnet API: ~2,880 calls/dayKey Insight
Self-hosted models have a fixed cost (GPU rental) regardless of volume, while API costs scale linearly. This means self-hosting is more expensive at low volumes but dramatically cheaper at high volumes. The breakeven point depends on the model, the GPU, and the specific task's token profile.
Finishing Up
Once you have completed all three exercises:
- Run all cells from top to bottom to verify everything executes without errors (Kernel → Restart & Run All).
- Save the notebook.
- Review your comparison matrix and cost projections — identify which model you would recommend for your chosen use case and why.
Self-Check
- ✓ Exercise 1: Your matrix includes at least 5 models and all 6 criteria.
- ✓ Exercise 1: Scores are justified — you can explain why each model got each score.
- ✓ Exercise 2: You tested at least 3 models on at least 3 prompts each.
- ✓ Exercise 2: You recorded both quality scores and latency for each call.
- ✓ Exercise 3: Your cost table covers at least 3 volume tiers.
- ✓ Exercise 3: You identified the crossover point for self-hosting vs. API.