Lab Guide

Format: Python script (lab.py)  ·  Time estimate: 45–60 minutes  ·  Prerequisites: Node.js 18+, Python 3.10+, API keys for Anthropic and/or OpenAI

Setup

This lab uses promptfoo, a Node.js CLI tool, as the primary evaluation framework. You will also write a small Python helper script.

Step 1: Install promptfoo

# Install promptfoo globally (requires Node.js 18+)
npm install -g promptfoo

# Verify installation
promptfoo --version

Step 2: Set up API keys

# Ensure your .env file has the required keys
# (or export them directly in your shell)
export ANTHROPIC_API_KEY="your-anthropic-key"
export OPENAI_API_KEY="your-openai-key"

Step 3: Create a working directory

# Create a fresh directory for this lab
mkdir -p lesson29-lab && cd lesson29-lab

Step 4: Install Python dependencies

pip install anthropic openai python-dotenv

Exercise 1: Create a promptfoo Configuration

Goal

Write a YAML config that evaluates a summarisation prompt against two LLM providers with five test cases.

Steps

1. Create a file called promptfooconfig.yaml in your working directory.
2. Define two providers:

   providers:
     - anthropic:messages:claude-sonnet-4-20250514
     - openai:gpt-4o

3. Define a summarisation prompt template:

   prompts:
     - |
       Summarise the following text in exactly 2 sentences.
       Be factual and concise. Do not add information not present in the text.
   
       Text: {{text}}
   
       Summary:

4. Add five test cases with input text and assertions. Here is one example — create four more with different input texts:

   tests:
     - vars:
         text: >
           The European Central Bank held interest rates steady at 4.5%
           on Thursday, pausing its aggressive tightening cycle for the
           first time in over a year. ECB President Christine Lagarde
           said inflation was falling but warned that rate cuts were
           not yet on the table.
       assert:
         - type: contains
           value: "interest rates"
         - type: contains
           value: "ECB"
         - type: not-contains
           value: "I'm sorry"
         - type: javascript
           value: "output.split('.').filter(s => s.trim()).length <= 3"

5. Run the evaluation:

   promptfoo eval

Expected Output

Eval complete.

| # | text (truncated)            | anthropic:messages:... | openai:gpt-4o |
|---|-----------------------------|-----------------------|---------------|
| 1 | The European Central Bank...| [PASS] 4/4 assertions | [PASS] 4/4    |
| 2 | ...                         | [PASS]                | [PASS]        |
| ...                                                                      |

Pass rate: anthropic 100%, openai 100%

Exercise 2: Add Advanced Assertions

Goal

Explore the full range of promptfoo assertion types by adding regex, JSON, and LLM-graded assertions.

Steps

1. Add a new test case that requires JSON output. Change the prompt for this test:

     - vars:
         text: >
           Apple reported Q3 revenue of $81.8 billion, up 5% year-over-year.
           iPhone revenue was $39.6 billion. Services revenue hit a record
           $21.2 billion. CEO Tim Cook called it a "remarkable quarter."
       assert:
         - type: is-json
         - type: javascript
           value: |
             const data = JSON.parse(output);
             return data.revenue !== undefined && data.company !== undefined;

   You will need a second prompt that asks the model to output JSON. Add it to the prompts array:

   prompts:
     - |
       Summarise the following text in exactly 2 sentences.
       Be factual and concise.
   
       Text: {{text}}
   
       Summary:
     - |
       Extract key financial data from the following text as JSON.
       Return an object with keys: company, revenue, highlights.
   
       Text: {{text}}
   
       JSON:

2. Add an LLM-graded assertion to one of your summarisation test cases:

       assert:
         - type: llm-rubric
           value: >
             The summary is factually accurate, contains no information
             not present in the original text, and is written in a
             neutral tone. It should be exactly 2 sentences long.

3. Add a regex assertion that checks for a date pattern in a test case involving dates:

         - type: regex
           value: "\\b\\d{4}\\b"

4. Run the evaluation again and compare results:

   promptfoo eval

Exercise 3: Provider Comparison

Goal

Compare Claude and GPT-4o side by side on the same test cases and analyse the differences in pass rate, output quality, and cost.

Steps

1. Ensure both providers are configured in your YAML (from Exercise 1).
2. Add a cost assertion to monitor spending:

         - type: cost
           threshold: 0.02

3. Run the evaluation and export results as JSON:

   promptfoo eval --output results.json

4. Write a Python script (this is your lab.py) that reads results.json and prints a comparison summary:

   """
   Lesson 29 Lab: Eval Framework Analysis
   ---------------------------------------
   TODO: Load results.json from a promptfoo eval run and print:
     1. Pass rate per provider
     2. Average response length per provider
     3. Total cost per provider
     4. Which provider "won" on each test case
   """
   import json
   from pathlib import Path
   
   def load_results(path: str = "results.json") -> dict:
       """Load promptfoo results JSON."""
       # TODO: Load and return the JSON data
       pass
   
   def compute_pass_rates(results: dict) -> dict:
       """Compute pass rate per provider."""
       # TODO: Iterate through results and compute pass/total for each provider
       pass
   
   def compute_avg_length(results: dict) -> dict:
       """Compute average output character length per provider."""
       # TODO: Iterate through results and compute average output length
       pass
   
   def print_comparison(results: dict):
       """Print a formatted comparison report."""
       # TODO: Call the functions above and print a formatted summary
       pass
   
   if __name__ == "__main__":
       print_comparison(load_results())

5. Run your analysis script:

   python lab.py

Expected Output

=== Provider Comparison Report ===

Provider                          | Pass Rate | Avg Length | Total Cost
----------------------------------|-----------|------------|----------
anthropic:messages:claude-sonnet.. | 92.0%     | 187 chars  | $0.034
openai:gpt-4o                     | 88.0%     | 203 chars  | $0.028

Test-level breakdown:
  Test 1: Both passed
  Test 2: Both passed
  Test 3: anthropic passed, openai FAILED (llm-rubric)
  Test 4: Both passed
  Test 5: Both passed

Exercise 4: Generate the HTML Report

Goal

Use promptfoo's built-in reporting to generate and interpret an interactive HTML eval report.

Steps

1. Open the promptfoo web viewer:

   promptfoo view

2. In the web UI, explore:
   • The results table — click any cell to see the full output
   • The assertion details — see which assertions passed and failed
   • The side-by-side view — compare outputs from different providers
3. Export the results as a shareable HTML file:

   promptfoo eval --output report.html

4. Open report.html in your browser and verify it contains the same data as the web UI.

Finishing Up

Once you have completed all four exercises:

1. Verify your promptfooconfig.yaml has at least 5 test cases with diverse assertion types.
2. Verify your lab.py script loads and analyses the eval results.
3. Compare your implementation with lab_answer.py.

Self-Check

• ✓ Exercise 1: Your YAML config defines 2 providers, 1+ prompts, and 5 test cases.
• ✓ Exercise 1: promptfoo eval runs without errors and shows a results table.
• ✓ Exercise 2: You used at least 4 different assertion types (contains, regex, is-json, llm-rubric).
• ✓ Exercise 3: Your Python script correctly computes pass rates and average lengths from the JSON output.
• ✓ Exercise 3: You can explain why one provider might score higher than another on the same test.
• ✓ Exercise 4: You can navigate the promptfoo web UI and find specific assertion failures.