Lab Guide

Format: Python script (lab.py)  ·  Time estimate: 45-60 minutes  ·  Prerequisites: Python 3.10+, anthropic, rouge-score, scipy, python-dotenv

Setup

Before starting, ensure your environment is ready:

# From the project root
pip install anthropic rouge-score scipy python-dotenv

# Ensure your .env file has your API key
# ANTHROPIC_API_KEY=your_key_here

Verify setup by running:

python -c "from rouge_score import rouge_scorer; print('rouge-score OK')"
python -c "from scipy import stats; print('scipy OK')"
python -c "import anthropic; print('anthropic OK')"

Exercise 1: Build an Eval Pipeline

Goal

Build a reusable evaluation harness that computes exact match, ROUGE-L, and a simple semantic check across a dataset of question-answer pairs.

Steps

1. Define a small evaluation dataset as a list of dictionaries, each containing input, expected, and category fields:

   EVAL_SET = [
       {
           "input": "What is the capital of France?",
           "expected": "Paris",
           "category": "factual"
       },
       {
           "input": "Summarise photosynthesis in one sentence.",
           "expected": "Photosynthesis is the process by which plants convert sunlight, water, and carbon dioxide into glucose and oxygen.",
           "category": "summarisation"
       },
       # Add 8-10 more examples across different categories
   ]

2. Implement an exact_match(predicted, expected) function that normalises both strings (lowercase, strip whitespace, remove punctuation) before comparing.
3. Implement a compute_rouge_l(predicted, expected) function using the rouge-score library:

   from rouge_score import rouge_scorer
   
   def compute_rouge_l(predicted: str, expected: str) -> float:
       scorer = rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True)
       scores = scorer.score(expected, predicted)
       return scores["rougeL"].fmeasure

4. Write a run_eval(eval_set, model_fn) function that:
   • Runs each input through model_fn to get a prediction
   • Computes exact match and ROUGE-L for each example
   • Returns a results list with per-example scores and aggregate statistics
5. Create a simple model_fn that calls the Claude API with a basic prompt:

   import anthropic
   from dotenv import load_dotenv
   
   load_dotenv()
   client = anthropic.Anthropic()
   
   def call_claude(prompt: str) -> str:
       response = client.messages.create(
           model="claude-sonnet-4-20250514",
           max_tokens=256,
           messages=[{"role": "user", "content": prompt}]
       )
       return response.content[0].text

6. Run the pipeline and print a summary table showing per-example and aggregate results.

Expected Output

=== Eval Results ===
 #  Category        Exact Match  ROUGE-L  Prediction (truncated)
 1  factual         1.0          1.00     Paris
 2  summarisation   0.0          0.62     Plants use sunlight to convert...
 3  factual         1.0          1.00     7
 ...

=== Aggregate ===
Exact Match:  0.40  (4/10)
ROUGE-L mean: 0.71
ROUGE-L std:  0.28

Exercise 2: LLM-as-Judge

Goal

Implement an LLM-as-judge evaluator with a structured rubric, then calibrate it against your own human labels.

Steps

1. Define a judge prompt template with a clear rubric:

   JUDGE_PROMPT = """You are an expert evaluator. Rate the following answer
   on a scale of 1-5 based on this rubric:
   
   5 - Fully correct, complete, and well-expressed
   4 - Correct with minor omissions or style issues
   3 - Partially correct, missing important details
   2 - Contains significant errors or is misleading
   1 - Incorrect, irrelevant, or harmful
   
   Question: {question}
   Reference answer: {reference}
   Model answer: {model_answer}
   
   Respond with ONLY valid JSON in this exact format:
   {{"score": , "reasoning": ""}}"""

2. Write a judge_answer(question, reference, model_answer) function that:
   • Formats the judge prompt
   • Calls Claude (use the same or a stronger model as judge)
   • Parses the JSON response
   • Returns the score and reasoning
3. Run the judge on all examples from Exercise 1. Store the judge scores alongside the automated metrics.
4. Human calibration: Manually score each model output yourself (1-5) before looking at the judge scores. Then compare:

   human_scores = [5, 4, 5, 3, ...]  # Your manual ratings
   judge_scores = [5, 4, 5, 4, ...]   # From the LLM judge
   
   # Agreement rate (within 1 point)
   agreement = sum(
       1 for h, j in zip(human_scores, judge_scores) if abs(h - j) <= 1
   ) / len(human_scores)
   print(f"Agreement (within 1 point): {agreement:.0%}")

Expected Output

=== LLM-as-Judge Results ===
 #  Judge Score  Reasoning
 1  5            Correct: the capital of France is Paris
 2  4            Accurate but less detailed than reference
 3  5            Exact correct numerical answer
 ...

=== Calibration ===
Human vs Judge agreement (within 1 point): 80%
Mean human score: 3.8
Mean judge score: 4.1
Judge bias: +0.3 (judge rates slightly higher)

Exercise 3: Custom Benchmark with A/B Testing

Goal

Create a focused eval set for a specific task, run two prompt variants on it, and determine whether the difference is statistically significant.

Steps

1. Choose a task domain. For this lab, use technical Q&A about Python. Create 20 question-answer pairs:

   PYTHON_EVAL = [
       {"input": "What does the `enumerate()` function return?",
        "expected": "enumerate() returns an iterator of tuples containing indices and values from an iterable."},
       {"input": "How do you handle a KeyError in Python?",
        "expected": "Use a try/except block: try: value = d[key] except KeyError: handle the missing key. Alternatively, use dict.get(key, default)."},
       # ... 18 more examples
   ]

2. Define two prompt variants — a baseline and a candidate:

   def prompt_baseline(question: str) -> str:
       return question
   
   def prompt_candidate(question: str) -> str:
       return f"""You are a Python expert. Answer the following question
   concisely and accurately. Include a brief code example if relevant.
   
   Question: {question}"""

3. Run both prompts on all 20 examples. Use the LLM-as-judge from Exercise 2 to score each output.
4. Compute the Wilcoxon signed-rank test to check for statistical significance:

   from scipy.stats import wilcoxon
   
   baseline_scores = [...]   # judge scores for baseline
   candidate_scores = [...]  # judge scores for candidate
   
   stat, p_value = wilcoxon(baseline_scores, candidate_scores)
   print(f"Wilcoxon p-value: {p_value:.4f}")
   print(f"Significant at p<0.05: {p_value < 0.05}")

5. Print a comparison report:

Expected Output

=== A/B Test Results ===
                  Baseline    Candidate
Mean judge score: 3.65        4.20
Median:           4           4
Std dev:          0.93        0.77

Wins / Ties / Losses: 12 / 5 / 3
Wilcoxon statistic: 23.0
p-value: 0.0127
Significant at p<0.05: Yes

Conclusion: The candidate prompt significantly outperforms
the baseline (p=0.013, mean improvement +0.55 points).

Finishing Up

Once you have completed all three exercises:

1. Run the full script end-to-end: python lab.py
2. Verify all three exercises produce output without errors.
3. Compare your implementation with lab_answer.py — focus on the judge prompt design and statistical testing.

Self-Check

• ✓ Exercise 1: Your pipeline computes both exact match and ROUGE-L for every example.
• ✓ Exercise 1: Aggregate statistics include mean and standard deviation.
• ✓ Exercise 2: Your judge returns structured JSON with score and reasoning.
• ✓ Exercise 2: You compared judge scores against your own human labels.
• ✓ Exercise 3: You ran a paired statistical test (not just compared means).
• ✓ Exercise 3: You can explain what the p-value means in context.