Lesson 29 — Eval Frameworks

Lesson 29 · Slide 1

Eval Frameworks

Systematic evaluation is what separates production-grade LLM applications from prototypes. This lesson covers the tooling landscape for measuring, comparing, and regression-testing LLM outputs.

Learning Objectives

Configure and run promptfoo evaluations with YAML configs, assertions, and multiple providers
Understand the purpose and design of OpenAI Evals, Ragas, DeepEval, and LangSmith
Build a CI/CD eval pipeline that catches prompt regressions before deployment
Compare frameworks on features, ease of use, and integration patterns

Lesson 29 · Slide 2

Why Eval Frameworks Exist

Manual spot-checking does not scale. As your prompt library grows, you need automated, reproducible evaluation to answer three questions:

Did my change improve things? — A/B comparison across prompt versions
Does it still work on edge cases? — regression testing on curated datasets
Which model is best for this task? — provider comparison at fixed cost

The Eval Loop

Define test cases → Run against providers → Apply assertions / graders → Report pass/fail + scores → Iterate on prompts

Notes: Draw parallels to unit testing in software engineering. Evals are the test suite for your prompts.

Lesson 29 · Slide 3

promptfoo: YAML-Driven Eval

promptfoo is an open-source CLI tool (Node.js) that lets you define evaluations in YAML and run them from the terminal or CI.

Core Concepts

Providers — which models to test (Claude, GPT-4, Llama, etc.)
Prompts — one or more prompt templates with {{variables}}
Tests — input variables + assertions (expected behaviour)
Assertions — contains, regex, JSON schema, LLM-graded, similarity, cost thresholds

# promptfooconfig.yaml
providers:
  - openai:gpt-4o
  - anthropic:messages:claude-sonnet-4-20250514

prompts:
  - "Summarise this article in 2 sentences: {{article}}"

tests:
  - vars:
      article: "The Federal Reserve held interest rates steady..."
    assert:
      - type: contains
        value: "interest rates"
      - type: llm-rubric
        value: "The summary is factually accurate and concise"
      - type: cost
        threshold: 0.01

Notes: Show the YAML config live and run promptfoo eval to demonstrate the output table.

Lesson 29 · Slide 4

promptfoo Assertions Deep Dive

Assertions are the heart of promptfoo. They define what "correct" means for each test case.

Assertion Type	What It Checks	Example
`contains`	Output includes a substring	`value: "Python"`
`not-contains`	Output must not include substring	`value: "I'm sorry"`
`regex`	Output matches a regex	`value: "\\d{4}-\\d{2}-\\d{2}"`
`is-json`	Output is valid JSON	(no value needed)
`javascript`	Custom JS function returns true	`value: "output.length < 500"`
`similar`	Cosine similarity above threshold	`value: "expected text", threshold: 0.8`
`llm-rubric`	An LLM judges the output	`value: "Answer is helpful and accurate"`
`cost`	API cost below threshold	`threshold: 0.02`

Combining Assertions

Each test can have multiple assertions. A test passes only if all assertions pass. Use weight to score partial credit when using --grading mode.

Lesson 29 · Slide 5

The Broader Landscape

promptfoo is not the only option. Here is how the major frameworks compare:

OpenAI Evals

Open-source framework from OpenAI
Python-based, YAML + code eval definitions
Strong on classification and factual benchmarks
Tightly coupled to OpenAI API by default

Ragas

Purpose-built for RAG evaluation
Metrics: faithfulness, answer relevancy, context recall, context precision
Generates synthetic test questions from your corpus
Python library, integrates with LangChain

DeepEval

Pytest-style interface: assert_test()
14+ built-in metrics (hallucination, bias, toxicity)
Runs as standard Python tests
Optional cloud dashboard for tracking

LangSmith

Tracing + evaluation platform by LangChain
Captures full chain execution traces
Run evals on traced data or custom datasets
Best when already using LangChain/LangGraph

Lesson 29 · Slide 6

Framework Comparison

Feature	promptfoo	OpenAI Evals	Ragas	DeepEval	LangSmith
Language	Node.js CLI	Python	Python	Python	Python + UI
Config format	YAML	YAML + code	Code	Code (pytest)	Code + UI
Multi-provider	Built-in	Limited	Any LLM	Any LLM	LangChain providers
RAG-specific	Partial	No	Yes (core focus)	Yes	Yes
CI/CD ready	Excellent	Good	Good	Excellent	Good
LLM-as-judge	llm-rubric	Model-graded	Built-in	Built-in	Built-in
Dashboard	Local web UI	None	None	Cloud (optional)	Cloud
License	MIT	MIT	Apache 2.0	Apache 2.0	Proprietary

Notes: Emphasise that the "best" framework depends on the team's stack and the evaluation task. promptfoo excels at prompt comparison; Ragas excels at RAG pipelines; DeepEval fits pytest workflows.

Lesson 29 · Slide 7

CI/CD Eval Pipelines

Treat prompts like code: version them, test them, gate deployments on eval results.

Pipeline Architecture

PR opened → CI runs promptfoo eval → assertions checked → pass/fail reported on PR → merge only if evals pass

# .github/workflows/eval.yml
name: Prompt Eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm install -g promptfoo
      - run: promptfoo eval --no-cache
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - run: promptfoo eval --output results.json
      - name: Check pass rate
        run: |
          PASS_RATE=$(node -e "const r=require('./results.json'); const s=r.results.stats; console.log((s.successes/s.total*100).toFixed(1))")
          echo "Pass rate: ${PASS_RATE}%"
          if (( $(echo "$PASS_RATE < 90" | bc -l) )); then exit 1; fi

Regression Testing for Prompts

Maintain a golden dataset of input/expected-output pairs
Run evals on every prompt change — catch regressions before they reach users
Track eval scores over time to detect gradual degradation (model updates, API changes)

Lesson 29 · Slide 8

Lab & Homework

Lab Exercises

Install & configure promptfoo — write a YAML config with two providers and five test cases
Assertion types — use contains, regex, is-json, and llm-rubric assertions
Provider comparison — run the same prompts against Claude and GPT-4o; compare pass rates and costs
Eval report — generate and interpret the promptfoo HTML report

Homework Tasks

Add a DeepEval test file that checks for hallucination and answer relevancy on the same test cases
Design a GitHub Actions workflow that runs promptfoo evals on pull requests
Write a comparison report (300 words) on when to use promptfoo vs. Ragas vs. DeepEval

📥 Lab Files

⬇ lab.py ⬇ lab_answer.py