Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Familiarity with LLM APIs (Lessons 01–18)

1. Why Model Selection Matters

The LLM landscape has matured rapidly. In 2023, most teams defaulted to "use the biggest model available." Today, there are dozens of viable models spanning a 100x cost range, and the right choice depends entirely on your specific use case. A team that blindly uses a flagship model for every task will either overpay dramatically or, worse, hit latency and throughput bottlenecks that a smaller model would have avoided entirely.

Model selection is not a one-time decision. As new models are released, pricing changes, and your application evolves, the optimal choice shifts. The goal of this lesson is to give you a repeatable framework for making and revisiting this decision.

2. The Six Selection Criteria

Every model selection decision can be decomposed into six dimensions. No model wins on all six, which is why the trade-offs matter.

Task Complexity

This is the most important criterion. Simple tasks — classification, extraction, formatting — can be handled by small, fast models with minimal quality loss. Complex tasks — multi-step reasoning, nuanced writing, code generation with edge cases — require frontier models. The mistake teams make most often is overestimating the complexity of their task. A support ticket classifier does not need Opus; Haiku will do the job at a fraction of the cost.

Latency Requirements

Latency has two components: time to first token (TTFT) and tokens per second (TPS) for generation. For interactive applications like chatbots, TTFT under 500ms is critical for a responsive user experience. For batch processing jobs, throughput (total tokens per minute) matters more than per-request latency. Smaller models are consistently faster: Haiku and GPT-4o mini typically achieve 2–5x the TPS of their larger siblings.

Cost Budget

API pricing follows a predictable pattern: larger, more capable models charge more per token, and output tokens cost more than input tokens (typically 3–5x). At low volumes (hundreds of calls per day), the cost difference between models is negligible. At high volumes (millions of calls per day), model selection becomes the single largest lever for cost control. A typical pattern: Opus at $75/M output tokens vs. Haiku at $4/M output tokens is an 18.75x difference — which translates to tens of thousands of dollars monthly at scale.

Privacy and Compliance

For regulated industries (healthcare, finance, government), data handling is often the binding constraint. Key questions: Can data be sent to a third-party API? Does the provider offer data processing agreements (DPAs)? Is the data used for training? Anthropic, OpenAI, and Google all offer enterprise tiers with contractual guarantees, but some organisations require on-premises deployment, which limits the choice to open-source models that can be self-hosted.

Multimodal Needs

If your application processes images, audio, or video alongside text, the model must support those modalities natively. Claude, GPT-4o, and Gemini all support vision (image input). Gemini 1.5 Pro additionally supports native audio and video input with its 1M-token context. For text-only tasks, multimodal capability is irrelevant — do not pay a premium for features you will not use.

Context Window Size

Context windows range from 8K tokens (older open-source models) to 1M tokens (Gemini 1.5 Pro). If your task involves processing long documents — legal contracts, codebases, research papers — you need a model with a sufficiently large window. However, larger context does not mean better: models still suffer from reduced accuracy on information placed in the middle of long contexts (the "lost in the middle" phenomenon from Lesson 01), and cost scales linearly with input length.

3. The Proprietary Landscape

The three major providers — Anthropic, OpenAI, and Google — each offer a tiered family of models. Understanding the tiers is the key to practical selection.

Anthropic (Claude)

Claude offers three tiers: Haiku (fast, cheap, ideal for high-volume simple tasks), Sonnet (the workhorse for most production applications, strong at coding and analysis), and Opus (the frontier model for complex reasoning and agentic workflows). All share a 200K context window. Claude is particularly strong at following complex instructions, handling long documents, and producing well-structured output.

OpenAI (GPT)

GPT-4o is the flagship multimodal model with strong all-round performance and a large ecosystem of tools and integrations. GPT-4o mini is one of the cheapest proprietary models available, making it an excellent choice for high-volume, lower-complexity tasks. The o1 and o3 reasoning models excel at mathematics and complex logic but have higher latency and cost due to their chain-of-thought approach.

Google (Gemini)

Gemini 1.5 Pro differentiates on context window size (1M tokens) and native multimodal support including audio and video. Gemini 1.5 Flash is extremely cost-efficient and fast. Google's models are well-suited for applications that require processing very long documents or multimedia content.

4. The Open-Source Option

Open-source models like Llama 3, Mistral, Mixtral, Qwen, and DeepSeek have closed much of the capability gap with proprietary models, especially for specific tasks. The decision to self-host involves a different kind of cost calculus.

Advantages: Full data control (nothing leaves your infrastructure), no per-token API cost after hardware investment, ability to fine-tune with full weight access, and no rate limits. Disadvantages: GPU infrastructure costs ($2–$10/hour for a single inference server), MLOps complexity, no SLA from a provider, and you are responsible for security patching and model updates.

The crossover point is typically around 10–50 million tokens per day: below that, APIs are cheaper when you factor in engineering time; above that, self-hosting becomes economically attractive. But this calculation is highly dependent on your team's MLOps maturity.

5. Understanding Benchmarks

Benchmarks are useful for creating an initial shortlist but should never be the sole basis for model selection. The most commonly cited benchmarks include:

MMLU (Massive Multitask Language Understanding): Tests knowledge across 57 academic subjects. Useful for general capability assessment, but the multiple-choice format does not reflect real-world generation tasks, and top models are saturating the benchmark.
HumanEval / MBPP: Test code generation ability. HumanEval contains only 164 problems, making it statistically noisy. Models can also be specifically optimised for these benchmarks.
GPQA (Graduate-Level Google-Proof Q&A): Tests expert-level reasoning. Useful for distinguishing frontier models but irrelevant if your task does not require PhD-level knowledge.
LMSYS Chatbot Arena: Uses human preference votes to generate Elo ratings. The most ecologically valid benchmark, but tends to reward verbosity and confident-sounding responses over conciseness and accuracy.

The fundamental limitation of all benchmarks is task mismatch: your production task — classifying insurance claims, generating product descriptions, answering questions about your codebase — probably looks nothing like any benchmark. The only reliable evaluation is testing candidate models on a representative sample of your actual data.

6. Model Routing and Cascading

The most sophisticated approach to model selection is not choosing one model at all — it is using multiple models in a routing or cascading architecture.

Routing uses a lightweight classifier (which could itself be a small LLM, a rules engine, or a traditional ML model) to examine each incoming request and direct it to the most cost-effective model that can handle it. For example, a customer support system might route "What are your business hours?" to Haiku and "I need to dispute a complex billing charge involving multiple subscriptions" to Sonnet.

Cascading takes a try-and-escalate approach: every request starts at the cheapest model. If the response fails a quality check (format validation, confidence score, or a lightweight verifier), the request is re-sent to a more capable model. In practice, if 80% of requests are handled by the cheap model, you achieve roughly 80% cost savings on those requests while maintaining quality parity on the remaining 20%.

Both patterns add engineering complexity. A poorly designed router wastes money by over-escalating, and a poorly designed cascade adds latency from failed attempts. Start simple: route by explicit categories first, measure, and add sophistication only when the data justifies it.

7. A Practical Decision Process

Here is a step-by-step process you can apply to any new model selection decision:

Define "good enough": Establish quality criteria using 50–100 representative examples from your real task. Have a human label the expected outputs.
Start with the best: Run your evaluation set through the strongest available model (e.g., Opus or GPT-4o). This sets the quality ceiling.
Test downward: Run the same evaluation on progressively cheaper models. For each, measure quality (against your labelled set), latency, and cost per task.
Find the knee: Plot quality vs. cost. There is usually a point where quality drops sharply — choose the cheapest model above that threshold.
Consider routing: If some inputs are obviously simple and others complex, a routing strategy may let you use a cheap model for the majority while reserving the expensive model for the hard cases.
Re-evaluate quarterly: New models and price changes can shift the optimal choice. Build your evaluation pipeline once and rerun it regularly.

Key Takeaways

Principle	Details
No single best model	The right model depends on your specific task, volume, latency, and compliance needs
Start strong, optimise down	Establish a quality baseline with a frontier model before testing cheaper alternatives
Benchmarks are a starting point	Always validate on your own data; benchmark scores do not predict production performance
Routing saves money at scale	Multi-model architectures can reduce cost 60–80% while maintaining quality
Re-evaluate regularly	The model landscape changes quarterly; build reusable eval pipelines