Homework

Task 1: Add BLEU Scoring and Compare Metrics

Goal: Extend your eval pipeline from the lab with BLEU scoring and analyse how different metrics correlate with human judgment.

Instructions

1. Install the nltk library and use nltk.translate.bleu_score.sentence_bleu to compute BLEU scores for each example in your eval set.
2. For each example, you should now have three automated metrics: exact match, ROUGE-L, and BLEU.
3. Manually rate each model output yourself on a 1-5 scale (your human judgment).
4. Compute the Spearman rank correlation between each automated metric and your human scores:

   from scipy.stats import spearmanr
   
   corr_rouge, p_rouge = spearmanr(rouge_scores, human_scores)
   corr_bleu, p_bleu = spearmanr(bleu_scores, human_scores)
   print(f"ROUGE-L vs Human: r={corr_rouge:.3f} (p={p_rouge:.3f})")
   print(f"BLEU vs Human:    r={corr_bleu:.3f} (p={p_bleu:.3f})")

5. Write 3-4 sentences analysing your findings. Which metric correlates better with your human judgment? Are there cases where one metric is misleading?

Deliverable

A script that prints a comparison table and your written analysis.

Self-check

• ✓ You computed BLEU, ROUGE-L, and human scores for at least 10 examples.
• ✓ You used Spearman correlation (not Pearson) — appropriate for ordinal data.
• ✓ Your analysis mentions at least one specific example where a metric was misleading.

Task 2: Benchmark Research Report

Goal: Deepen your understanding of how the LLM community measures progress by researching a benchmark not covered in the slides.

Instructions

Choose one of the following benchmarks (or another of your choosing):

• MATH — competition-level mathematics
• ARC (AI2 Reasoning Challenge) — science exam questions
• WinoGrande — commonsense coreference resolution
• BBH (BIG-Bench Hard) — tasks where LLMs initially underperformed humans
• MT-Bench — multi-turn conversation quality
• SWE-bench — real-world software engineering tasks

Write a 200-word analysis covering:

1. What it tests: What capability or skill does this benchmark measure?
2. Format: How are questions structured? How is scoring done?
3. Current state-of-the-art: Which models lead and what scores do they achieve?
4. Limitations: What does this benchmark fail to capture? Is there evidence of contamination?

Deliverable

A 200-word written analysis (plain text or in a comment block in your script).

Self-check

• ✓ You addressed all four points (what, format, SOTA, limitations).
• ✓ You cited at least one specific score or result.
• ✓ Your limitations section goes beyond "benchmarks are imperfect" — it is specific to the benchmark you chose.

Task 3: Design an Eval Strategy

Goal: Practice designing a complete evaluation strategy for a realistic LLM application.

Instructions

Choose one of the following scenarios (or use a real project you are working on):

• Customer support chatbot that answers questions about a SaaS product
• Code review assistant that provides feedback on pull requests
• Medical document summariser that extracts key findings from radiology reports

Design a complete eval strategy by specifying:

1. Eval dimensions: What aspects of quality matter? (accuracy, completeness, safety, tone, latency, etc.) Rank them by importance.
2. Automated metrics: Which automated metrics would you use for each dimension? Justify each choice.
3. LLM-as-judge: Write the exact rubric you would use. Define each score level (1-5) with concrete criteria specific to your chosen scenario.
4. Human eval protocol: How many examples? How many raters? What agreement threshold? How often do you re-evaluate?
5. Eval set composition: How would you source examples? What categories and difficulty levels would you include? How many examples per category?
6. Regression testing: How would you detect if a prompt change or model update degrades performance?

Deliverable

A structured document (in your script as comments, or a separate text file) covering all six points. Target 300-500 words total.

Self-check

• ✓ You addressed all six points with specifics, not generalities.
• ✓ Your rubric has concrete, scenario-specific criteria at each score level.
• ✓ Your eval set composition includes both normal and adversarial examples.
• ✓ Your regression testing plan specifies a concrete threshold or process.