Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Familiarity with LLM APIs, basic statistics

1. The Evaluation Problem

In traditional software engineering, testing is straightforward: given input X, the function should return Y. If it does, the test passes. LLM evaluation breaks this model in fundamental ways. A summarisation system might produce ten different summaries of the same article, all of which are correct. A code generation model might solve a problem with an approach you never anticipated. A chatbot response might be factually accurate but tonally wrong for your audience.

This makes evaluation the single most important — and most underinvested — part of LLM application development. Without rigorous eval, you are flying blind: you cannot tell whether a prompt change helped or hurt, whether a cheaper model is "good enough," or whether your system is regressing over time.

The good news is that evaluation is a solvable problem. It requires combining multiple approaches — automated metrics, model-based judgment, and targeted human review — into a layered strategy that matches the needs of your specific application.

2. Automated Metrics

Automated metrics are the foundation of any eval pipeline. They are fast, deterministic, and free. Their weakness is that they capture surface-level similarity rather than semantic correctness.

Exact Match

The simplest metric: does the output exactly equal the reference? This works well for classification, factual extraction, and code that must produce a specific output. It fails for any task where multiple valid outputs exist. A common variant is normalised exact match, which lowercases and strips whitespace before comparing.

BLEU (Bilingual Evaluation Understudy)

Originally designed for machine translation, BLEU measures n-gram overlap between the model output (candidate) and one or more reference texts. It computes precision for n-grams of size 1 through 4 and combines them with a geometric mean. A brevity penalty discourages outputs that are shorter than the reference.

BLEU scores range from 0 to 1 (often reported as 0-100). A score of 0.3-0.4 is considered decent for translation. However, BLEU has significant limitations: it penalises valid paraphrases, is insensitive to word order beyond the n-gram window, and correlates poorly with human judgment on many generative tasks.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

ROUGE is the standard metric for summarisation. While BLEU emphasises precision (how much of the candidate appears in the reference), ROUGE emphasises recall (how much of the reference appears in the candidate). The most commonly used variant is ROUGE-L, which measures the longest common subsequence between candidate and reference, capturing sentence-level structure without requiring exact n-gram matches.

Perplexity

Perplexity measures how "surprised" a language model is by a piece of text. Formally, it is the exponentiated average negative log-likelihood per token. Lower perplexity means the model considers the text more likely. Perplexity is useful for comparing language models on the same test corpus or measuring fluency, but it does not measure factual accuracy, task completion, or usefulness.

The Paraphrase Problem

Consider two summaries of the same article: "The CEO resigned after the scandal" and "Following the controversy, the company's top executive stepped down." These are semantically equivalent, but BLEU scores near zero because they share almost no n-grams. This is why surface-level metrics must always be supplemented with semantic evaluation.

3. LLM-as-Judge

The most important development in LLM evaluation over the past two years is using LLMs themselves as evaluators. The approach is simple: send the original prompt, the model's output, and a scoring rubric to a strong model, and ask it to rate the output.

This works surprisingly well. Research from LMSYS (the organisation behind the Chatbot Arena leaderboard) has shown that strong LLM judges achieve 80-85% agreement with human evaluators — comparable to human-human agreement rates. The advantages are compelling: LLM-as-judge is 10-100x cheaper than human evaluation, runs in seconds rather than days, and scales to thousands of examples trivially.

Designing a Good Rubric

The quality of LLM-as-judge evaluation depends entirely on the rubric. A vague rubric ("rate quality from 1-5") produces inconsistent results. An effective rubric:

Defines each score level with concrete criteria and examples
Separates independent dimensions (accuracy, completeness, style) rather than collapsing them into a single score
Specifies what constitutes a failure — hallucination, off-topic response, format violation
Asks the judge to provide reasoning before the score (chain-of-thought improves consistency)

Known Biases

LLM judges exhibit systematic biases that you must account for. Position bias: in pairwise comparisons, judges tend to prefer whichever answer appears first. Mitigate by running each comparison twice with swapped order. Verbosity bias: judges tend to prefer longer, more detailed answers even when the shorter answer is more appropriate. Self-enhancement bias: a model may rate its own outputs higher than those from other models. Use a different model family as the judge when possible.

4. Human Evaluation

Human evaluation remains the gold standard for subjective quality assessment, but it requires careful protocol design to be useful.

The two main approaches are absolute rating (each output is scored independently on a scale) and pairwise comparison (raters choose which of two outputs is better). Pairwise comparison is generally more reliable because humans are better at comparing two things than assigning absolute scores. The downside is that comparing N systems requires O(N^2) comparisons.

The critical metric for human eval is inter-annotator agreement. If your raters disagree frequently, it usually means your rubric is ambiguous, not that your raters are incompetent. Cohen's Kappa measures agreement between two raters, correcting for chance agreement. A Kappa above 0.6 indicates substantial agreement; below 0.4 suggests the rubric needs revision.

5. Benchmark Suites

The LLM community has developed standard benchmarks for comparing models. Understanding what they measure — and what they miss — is essential for interpreting model comparisons.

MMLU (Massive Multitask Language Understanding) tests knowledge across 57 academic subjects using multiple-choice questions. It measures breadth of knowledge but not depth, reasoning, or generation quality. HumanEval consists of 164 Python programming problems and measures code generation via pass@k (the probability that at least one of k generated solutions passes all test cases). GSM8K contains 8,500 grade-school math word problems requiring multi-step reasoning. HellaSwag tests commonsense reasoning by asking models to complete descriptions of everyday activities. TruthfulQA specifically tests whether models reproduce common misconceptions — a score of 100% would mean the model never gives a confidently wrong answer to a tricky question.

The key limitation of all benchmarks is contamination: benchmark questions may appear in training data, inflating scores. Additionally, benchmark performance may not correlate with performance on your specific task. Always treat benchmarks as a rough filter, not a definitive ranking.

6. Building Custom Eval Sets

For production applications, custom eval sets matter more than benchmarks. A good eval set is a curated collection of inputs, expected outputs (or scoring criteria), and metadata that reflects the actual distribution of queries your system handles.

Start by sampling 50-200 examples from real production data. Stratify by difficulty: include straightforward cases that any reasonable system should handle, challenging cases that test edge conditions, and adversarial cases designed to expose known failure modes. Have domain experts write reference answers or evaluation criteria for each example. Version your eval set in source control and never modify examples in place — add new ones and retire old ones.

7. A/B Testing and Statistical Significance

When comparing two systems (different prompts, models, or parameters), you need statistical rigour to know whether the difference is real or due to random variation. Because both systems are evaluated on the same inputs, use paired statistical tests. For binary outcomes (correct/incorrect), McNemar's test is appropriate. For ordinal scores, use the Wilcoxon signed-rank test.

Set your significance threshold (alpha, typically 0.05) and minimum detectable effect size before running the experiment. Report confidence intervals alongside p-values — a statistically significant but tiny improvement may not justify the cost of switching systems.

8. Eval-Driven Development

The most effective workflow for LLM applications is eval-driven development: define your evaluation criteria first, build the eval pipeline, then iterate on prompts and models until the eval passes. This mirrors test-driven development in software engineering. Every prompt change, model swap, or parameter tweak should be validated against the eval set before deployment. Over time, your eval set becomes the most valuable asset in your LLM project — more valuable than any individual prompt.