Lesson 19 · Slide 1
Choosing Models
A practical decision framework for selecting the right LLM for any task — balancing capability, cost, latency, and compliance.
Learning Objectives
- Apply a structured decision framework to select the best model for a given task
- Compare Claude, GPT, Gemini, and open-source model families across key dimensions
- Understand model routing and cascading as cost-performance optimisation strategies
- Interpret benchmarks (MMLU, HumanEval, etc.) and recognise their limitations
- Calculate cost per task and make data-driven model decisions
Lesson 19 · Slide 2
The Selection Framework
No single model is best for every task. Model selection is a multi-dimensional optimisation problem across six key criteria.
Capability Criteria
- Task complexity — Simple extraction vs. multi-step reasoning
- Multimodal needs — Text-only, vision, audio, video
- Context window — How much input the model must process at once
Operational Criteria
- Latency requirements — Real-time chat vs. batch processing
- Cost budget — Per-call cost at your expected volume
- Privacy / compliance — Data residency, on-premises needs, HIPAA/GDPR
Notes: Walk through a real example: "I need to classify support tickets" — low complexity, high volume, latency-sensitive. That immediately narrows the field to fast, cheap models.
Lesson 19 · Slide 3
Model Landscape: Proprietary
The three major proprietary providers each offer a family of models spanning the cost-capability spectrum.
| Family | Model | Context | Input $/M | Output $/M | Strengths |
|---|
| Claude | Haiku 3.5 | 200 K | $0.80 | $4.00 | Speed, cost, structured output |
| Claude | Sonnet 4 | 200 K | $3.00 | $15.00 | Balanced power, coding, analysis |
| Claude | Opus 4 | 200 K | $15.00 | $75.00 | Complex reasoning, agentic tasks |
| OpenAI | GPT-4o | 128 K | $2.50 | $10.00 | Multimodal, broad ecosystem |
| OpenAI | GPT-4o mini | 128 K | $0.15 | $0.60 | Ultra-low cost, fast |
| Google | Gemini 1.5 Pro | 1 M | $1.25 | $5.00 | Massive context, multimodal |
| Google | Gemini 1.5 Flash | 1 M | $0.075 | $0.30 | Speed, lowest cost |
Notes: Prices change frequently. The relative positioning matters more than exact numbers. Always check the provider's current pricing page before making production decisions.
Lesson 19 · Slide 4
Model Landscape: Open Source
Open-source models offer full control over deployment, data privacy, and per-token cost at scale — at the cost of operational complexity.
Leading Families
- Llama 3 (Meta) — 8B, 70B, 405B params; strong general-purpose performance
- Mistral / Mixtral — MoE architecture; excellent efficiency at inference
- Qwen 2.5 (Alibaba) — Strong multilingual and coding; up to 72B
- DeepSeek V3 — 671B MoE; competitive with top proprietary models
When to Choose Open Source
- Data sovereignty — Data cannot leave your infrastructure
- High volume — Self-hosting is cheaper above ~10 M tokens/day
- Fine-tuning — Full weight access for domain adaptation
- Latency control — Deploy on your own GPU fleet for predictable performance
The Hidden Costs
Self-hosting requires GPU infrastructure, MLOps expertise, monitoring, and ongoing maintenance. A "free" model can easily cost more than an API when you factor in GPU rental, engineering time, and uptime guarantees.
Lesson 19 · Slide 5
Benchmarks & Their Limitations
Benchmarks provide a common language for comparing models, but they are not the whole story.
| Benchmark | Measures | Limitations |
|---|
| MMLU | Multi-task knowledge (57 subjects) | Multiple-choice format; saturating at top end |
| HumanEval | Python code generation (164 problems) | Small dataset; only tests function-level code |
| GPQA | Graduate-level science Q&A | Narrow domain coverage |
| MT-Bench | Multi-turn conversation quality | LLM-as-judge introduces bias |
| LMSYS Chatbot Arena | Human preference (Elo ratings) | Favours verbose, confident answers |
Why Benchmarks Are Not Enough
- Benchmark contamination: Training data may include benchmark questions
- Task mismatch: Your production task may not resemble any benchmark
- Missing dimensions: Benchmarks rarely measure latency, cost, or reliability
- Rule of thumb: Use benchmarks to create a shortlist, then run your own evals
Lesson 19 · Slide 6
Model Routing & Cascading
Instead of picking one model, use multiple models strategically to optimise the cost-quality trade-off.
Routing
A classifier examines each request and sends it to the most appropriate model based on complexity.
- Simple queries → Haiku / GPT-4o mini
- Medium complexity → Sonnet / GPT-4o
- Hard problems → Opus / o1
Cascading
Start with a cheap model. If the response fails a quality check, retry with a stronger model.
- Try Haiku first (fast, cheap)
- Validate the output (format, confidence)
- Escalate to Sonnet only when needed
async def cascading_call(prompt: str) -> str:
"""Try cheap model first, escalate if quality check fails."""
# Stage 1: fast, cheap model
response = await call_model("claude-3-5-haiku-20241022", prompt)
if passes_quality_check(response):
return response # ~80% of requests stop here
# Stage 2: escalate to stronger model
return await call_model("claude-sonnet-4-20250514", prompt)
Notes: In production, cascading can reduce average cost by 60-80% while maintaining quality. The key is designing a reliable quality check — confidence scores, format validation, or a lightweight verifier model.
Lesson 19 · Slide 7
Cost-Performance Optimisation
The goal is not the cheapest model or the best model — it is the best model per dollar for your specific task.
The Cost-Performance Matrix
| Strategy | Impact | Effort |
|---|
| Right-size the model — use Haiku for simple tasks | 5–20x cost reduction | Low |
| Prompt optimisation — reduce token count | 10–40% savings | Medium |
| Caching — cache repeated prompts / prefixes | Up to 90% on repeat queries | Medium |
| Routing / cascading — multi-model strategy | 60–80% average cost reduction | High |
| Fine-tuning a small model — distil from a large model | 10–100x cheaper at inference | High |
Decision Heuristic
Start with the strongest model to establish a quality baseline. Then systematically try cheaper alternatives and measure whether quality is acceptable. Never optimise cost before you know what "good enough" looks like.
Lesson 19 · Slide 8
Lab & Homework
Lab Exercises
- Model Comparison Matrix — Build a structured comparison of 5+ models across the six selection criteria for a given use case
- Cross-Provider Testing — Send the same prompts to Claude, GPT, and Gemini; score quality, latency, and cost
- Cost-Per-Task Calculator — Calculate actual cost per task across models at realistic volumes
Homework Tasks
- Design a model routing strategy for a customer support system handling 50 K tickets/day across 3 complexity tiers
- Research a benchmark leaderboard; identify where the top model on the benchmark fails on a practical task
- Write a one-page recommendation memo choosing a model for a specific business scenario