Eval Fundamentals

How do you know if your LLM application is actually working? Evaluation is the discipline of measuring model output quality — and it is harder than it sounds.

Learning Objectives

• Understand why LLM evaluation is hard (subjectivity, open-endedness, distribution shift)
• Apply automated metrics: BLEU, ROUGE, perplexity, exact match
• Implement LLM-as-judge evaluation using a strong model to score outputs
• Design human evaluation protocols with inter-annotator agreement
• Navigate major benchmark suites: MMLU, HumanEval, GSM8K, HellaSwag, TruthfulQA
• Build custom eval sets and run A/B tests with statistical significance

Why Eval Is Hard

Evaluating LLM outputs is fundamentally different from evaluating traditional software. A unit test has a single expected answer. An LLM response can be correct in many different ways — or wrong in subtle ways that only a domain expert would catch.

Core Challenges

• Subjectivity: "Good" depends on the reader, the task, and the context. Two humans may disagree on which summary is better.
• Open-endedness: There is no single correct answer for most generative tasks. A translation can be faithful but awkward, or fluent but inaccurate.
• Distribution shift: A model that scores well on benchmarks may fail on your specific domain data.
• Metric gaming: Models (and prompts) can be optimised for a metric without improving actual quality.
• Scale: You cannot manually read every output in production. Automated metrics are necessary but imperfect.

Automated Metrics

Automated metrics provide fast, reproducible signals. None is perfect alone — use them in combination.

LLM-as-Judge

Use a strong model (e.g., Claude Sonnet) to evaluate outputs from a weaker or differently-prompted model. This is the most practical approach for evaluating open-ended generation at scale.

How It Works

1. Define a rubric — what constitutes a good answer (accuracy, completeness, tone, format)
2. Send the original prompt + model output + rubric to the judge model
3. Ask the judge to return a structured score (e.g., 1-5) with reasoning
4. Calibrate by comparing judge scores against human labels on a sample

judge_prompt = f"""Rate the following answer on a scale of 1-5.
Rubric:
- 5: Accurate, complete, well-structured
- 3: Partially correct, missing key details
- 1: Incorrect or irrelevant

Question: {question}
Answer: {model_output}

Return JSON: {{"score": int, "reasoning": str}}"""

Human Evaluation

Human eval remains the gold standard — but it is expensive, slow, and surprisingly unreliable without careful protocol design.

Key Protocols

Inter-Annotator Agreement

• Cohen's Kappa: measures agreement between two raters beyond chance. Kappa > 0.6 is "substantial" agreement.
• Krippendorff's Alpha: generalises to multiple raters and different scale types.
• Low agreement usually means your rubric is ambiguous, not that your raters are bad.

Benchmark Suites

Standard benchmarks let you compare models on well-defined tasks. Know what each measures — and what it does not.

Building Custom Evals & A/B Testing

The most impactful eval work is building evaluation sets specific to your application.

Custom Eval Set Design

1. Collect real examples — sample from production queries, not synthetic data
2. Stratify by difficulty — include easy, medium, and hard cases; include edge cases
3. Write gold references — have domain experts create reference answers
4. Target 50-200 examples — enough for statistical power, small enough to maintain
5. Version your eval set — track changes over time; never modify in place

A/B Testing and Statistical Significance

• Compare two prompt variants or model versions on the same eval set
• Use paired tests (McNemar's test for binary, Wilcoxon signed-rank for scores) because both systems see the same inputs
• Set significance threshold (typically p < 0.05) before running the test
• Report confidence intervals, not just pass/fail — a 2% improvement with a wide CI is not actionable

Lab & Homework