Homework

Task 1: DeepEval Regression Test

Goal: Write a pytest-compatible test file using DeepEval that checks for hallucination and answer relevancy on the same summarisation test cases from the lab.

Instructions

1. Install DeepEval:

   pip install deepeval

2. Create a file called test_eval.py that defines at least 3 test functions, each testing a different input text.
3. Each test should:
   • Call your chosen LLM API to generate a summary
   • Create an LLMTestCase with the input, output, and context
   • Assert using at least two metrics: AnswerRelevancyMetric and HallucinationMetric
4. Example skeleton:

   import pytest
   from deepeval import assert_test
   from deepeval.test_case import LLMTestCase
   from deepeval.metrics import AnswerRelevancyMetric, HallucinationMetric
   
   def get_summary(text: str) -> str:
       """Call your LLM to generate a summary."""
       # TODO: implement API call
       pass
   
   def test_ecb_summary():
       text = "The European Central Bank held interest rates steady..."
       output = get_summary(text)
       test_case = LLMTestCase(
           input=f"Summarise: {text}",
           actual_output=output,
           context=[text]
       )
       relevancy = AnswerRelevancyMetric(threshold=0.7)
       hallucination = HallucinationMetric(threshold=0.5)
       assert_test(test_case, [relevancy, hallucination])
   
   # TODO: Add test_apple_earnings() and test_climate_report()

5. Run the tests:

   pytest test_eval.py -v

Self-check

• ✓ You have at least 3 test functions in test_eval.py.
• ✓ Each test uses both AnswerRelevancyMetric and HallucinationMetric.
• ✓ All tests pass when run with pytest -v.
• ✓ You can explain what each metric measures and why the thresholds you chose are appropriate.

Task 2: GitHub Actions Eval Workflow

Goal: Design a CI/CD workflow file that runs promptfoo evaluations automatically on pull requests.

Instructions

1. Create a file called .github/workflows/prompt-eval.yml (you can create this locally; you do not need to push it to GitHub).
2. The workflow should:
   • Trigger on pull requests to the main branch
   • Set up Node.js 20 and install promptfoo
   • Run promptfoo eval --no-cache --output results.json
   • Parse the results JSON and fail the workflow if the overall pass rate is below 90%
   • Use GitHub secrets for API keys (reference them as ${{ secrets.ANTHROPIC_API_KEY }})
3. Add a comment in the workflow file explaining each step.
4. Bonus: add a step that posts the pass rate as a comment on the PR using gh pr comment.

Self-check

• ✓ The YAML is valid (test with python -c "import yaml; yaml.safe_load(open('.github/workflows/prompt-eval.yml'))").
• ✓ API keys are referenced from secrets, not hardcoded.
• ✓ The workflow fails if pass rate is below 90%.
• ✓ You can explain why --no-cache is important in CI (hint: reproducibility).

Task 3: Framework Comparison Report

Goal: Write a 300-word comparison of when to use promptfoo, Ragas, and DeepEval.

Instructions

Based on the theory reading and your hands-on experience in the lab, write a short report (~300 words) that addresses:

1. Use case fit: For each framework, describe the ideal scenario where it is the best choice. Be specific — name the type of application, team composition, and evaluation needs.
2. Trade-offs: What does each framework do poorly or not at all? What would you need to supplement it with?
3. Your recommendation: If you could only pick one framework for a team building a production LLM application that includes both a chatbot and a RAG-powered knowledge base, which would you choose and why? Would you combine frameworks?

Self-check

• ✓ Your report is 250–350 words.
• ✓ You discuss all three frameworks (promptfoo, Ragas, DeepEval).
• ✓ Your recommendation is justified with specific reasons, not just preference.
• ✓ You acknowledge trade-offs rather than declaring one framework universally "best".