Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Familiarity with LLM APIs, basic YAML, command-line tools

1. The Evaluation Problem

Building an LLM application is straightforward. Knowing whether it works well is not. Unlike traditional software where a function either returns the correct value or it does not, LLM outputs are probabilistic, nuanced, and subjective. A summarisation prompt might produce five different summaries that are all acceptable — or one that is fluent but factually wrong.

Manual review works for prototypes. You read a few outputs, adjust the prompt, and try again. But this approach breaks down as soon as you have more than a handful of test cases, more than one prompt, or more than one model to compare. You need automated, reproducible evaluation.

Eval frameworks solve this by providing a structured way to define what "good" looks like, run your prompts against that definition at scale, and report the results in a way that supports decision-making.

2. promptfoo: The Swiss Army Knife

promptfoo is an open-source, Node.js-based CLI tool that has become one of the most popular evaluation frameworks in the LLM ecosystem. Its strength is simplicity: you define everything in a single YAML file.

Configuration Structure

A promptfoo config file (promptfooconfig.yaml) has three main sections:

Providers — the models you want to test. promptfoo supports OpenAI, Anthropic, Google, Ollama, and any OpenAI-compatible endpoint out of the box.
Prompts — one or more prompt templates. Variables are inserted using {{double_braces}} syntax. You can reference external files with file://prompt.txt.
Tests — an array of test cases. Each test specifies input variables and one or more assertions that the output must satisfy.

Assertion Types

Assertions are where promptfoo shines. The framework supports deterministic checks (contains, regex, is-json, javascript), statistical checks (similar using embedding cosine similarity), and LLM-graded checks (llm-rubric where a judge model evaluates the output against a natural-language rubric).

The llm-rubric assertion type deserves special attention. It sends the output to a grading model (typically GPT-4o or Claude) along with your rubric text, and the grader returns a pass/fail judgement with reasoning. This is the closest you can get to human evaluation at scale, though it inherits the biases and limitations of the grading model.

Running and Reporting

Running promptfoo eval executes all test cases against all providers, applies all assertions, and produces a summary table showing pass rates per provider. The promptfoo view command opens a local web UI for exploring results interactively. Results can be exported as JSON for integration with other tools.

3. OpenAI Evals

OpenAI Evals is an open-source Python framework originally built for evaluating OpenAI models, though it can be adapted to other providers. It predates promptfoo and influenced the design of many subsequent frameworks.

Evals defines evaluation tasks as Python classes that inherit from a base evaluator. Each eval specifies a dataset (typically JSONL), a completion function (the model call), and a grading function. The framework includes a registry of pre-built evals covering common tasks like multiple-choice QA, classification, and translation.

The main advantages of OpenAI Evals are its large library of pre-built benchmarks and its integration with OpenAI's platform. The main disadvantage is that multi-provider testing requires more custom code than with promptfoo, and the setup is heavier for simple use cases.

4. Ragas: RAG-Specific Evaluation

If you are building a Retrieval-Augmented Generation (RAG) system, generic evaluation frameworks may not capture the failure modes that matter most. Ragas (Retrieval Augmented Generation Assessment) is a Python library purpose-built for evaluating RAG pipelines.

Ragas defines four core metrics:

Faithfulness — does the generated answer contain only information supported by the retrieved context? This catches hallucination, the most dangerous RAG failure mode.
Answer relevancy — is the answer actually relevant to the question asked, or did the model go off-topic?
Context precision — of the chunks retrieved, how many were actually relevant? High precision means your retrieval is not pulling in noise.
Context recall — of the chunks that should have been retrieved, how many were? High recall means you are not missing critical information.

A powerful feature of Ragas is synthetic test generation. Given your document corpus, Ragas can automatically generate question-answer pairs with known ground truth, giving you a test dataset without manual curation. This is invaluable for bootstrapping evaluation on a new RAG system.

5. DeepEval: Pytest for LLMs

DeepEval takes a different approach to the evaluation problem: it integrates directly with pytest, the standard Python testing framework. If your team already writes Python tests, DeepEval feels natural.

A typical DeepEval test looks like this:

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, HallucinationMetric

def test_summarisation():
    test_case = LLMTestCase(
        input="Summarise the Q3 earnings report",
        actual_output=model_output,
        context=["Revenue grew 12%...", "Operating margin was 23%..."]
    )
    relevancy = AnswerRelevancyMetric(threshold=0.7)
    hallucination = HallucinationMetric(threshold=0.5)
    assert_test(test_case, [relevancy, hallucination])

DeepEval ships with over 14 built-in metrics including hallucination detection, bias detection, toxicity scoring, and answer relevancy. Each metric uses an LLM judge internally and returns a score between 0 and 1. You set thresholds, and the test passes or fails like any other pytest assertion.

The pytest integration means DeepEval works with existing CI/CD infrastructure, IDE test runners, and test reporting tools with zero additional setup.

6. LangSmith: Tracing Meets Evaluation

LangSmith, built by the LangChain team, approaches evaluation from the observability angle. Before you can evaluate, you need to see what your application is actually doing — which chains fired, what was retrieved, what the model received, and what it returned.

LangSmith captures full execution traces of LangChain and LangGraph applications, storing them in a cloud dashboard. From there, you can create datasets from production traces, define evaluators (custom Python functions or LLM-graded), and run evaluations that score each trace.

The key advantage of LangSmith is the tight feedback loop between observability and evaluation. You can spot a bad output in production traces, add it to your eval dataset, write an assertion that catches it, and verify the fix — all within the same platform. The main limitation is that it works best within the LangChain ecosystem; teams not using LangChain may find the tracing integration less valuable.

7. Building CI/CD Eval Pipelines

The ultimate goal is to treat prompts with the same rigour as code. This means version-controlling your prompts, running automated evaluations on every change, and gating deployments on eval results.

A practical CI/CD eval pipeline has three stages:

Pre-commit — fast, deterministic checks. Does the prompt template parse? Are all required variables present? Does the output schema validate? These run in seconds with no API calls.
PR gate — full eval suite. Run promptfoo or DeepEval against your golden dataset. Require a minimum pass rate (e.g., 90%) to merge. This costs money (API calls) but catches regressions before they reach production.
Post-deploy monitoring — ongoing evaluation on production traffic. Sample a percentage of real requests, run them through your eval suite, and alert if scores drop below thresholds. LangSmith excels here.

Cost Awareness

Running evals costs money — every test case is an API call (or multiple, if using LLM-graded assertions). A 100-test-case suite with LLM grading against two providers means 400+ API calls per eval run. Budget for this and consider caching strategies for development iterations.

8. Choosing the Right Framework

There is no single "best" framework. The right choice depends on your context:

Quick prompt comparison across models → promptfoo. Minimal setup, YAML-based, excellent CLI output.
RAG pipeline evaluation → Ragas. Purpose-built metrics that capture RAG-specific failure modes.
Python-first teams with existing pytest infrastructure → DeepEval. Native integration, familiar patterns.
LangChain applications needing observability + eval → LangSmith. Tracing and evaluation in one platform.
Contributing to standardised benchmarks → OpenAI Evals. Large registry of existing benchmarks.

Many teams use more than one. A common pattern is promptfoo for prompt development and provider comparison, plus DeepEval or Ragas in the CI pipeline for regression testing, plus LangSmith for production monitoring.