How to Run Evals on Your LLM App With Promptfoo

You changed a system prompt, skimmed two outputs, and shipped it. That is how most prompt regressions reach production. An eval closes that gap: you write test cases with pass/fail assertions once, then run promptfoo eval to score every prompt and model against them before you merge. This guide takes you from an empty folder to a passing eval wired into continuous integration, in about fifteen minutes.

The discipline is spreading. Promptfoo, the open-source tool this guide uses, reported 130,000 monthly active users and adoption inside 25% of the Fortune 500 in its March 2026 announcement that it is joining OpenAI. The tool stays open source under its existing license.

Key Takeaways

The short answer: install with npx, define your prompts, providers, and test cases in promptfooconfig.yaml, run promptfoo eval, then open the web view to compare results.
Assertions come in two families: deterministic checks (contains, is-json, regex, cost, latency) that run in code, and model-graded checks (llm-rubric, factuality) that use an LLM to judge.
Model-graded assertions need their own grading provider. Set it once with defaultTest so every rubric uses a model your key can actually call.
In CI, a failing assertion returns a non-zero exit code, so a prompt regression fails the build instead of reaching users.
The tool is provider-agnostic. One config compares OpenAI, Anthropic, and local models side by side.
OpenAI agreed to acquire Promptfoo in March 2026; it remains open source, so nothing in this guide depends on the deal.

What You Need Before You Start

Three things, no account required to begin.

Node.js 18 or newer, which gives you npx. Nothing gets installed globally.
An API key for at least one provider, exported in the shell you will run the eval from (for example OPENAI_API_KEY or ANTHROPIC_API_KEY).
A prompt worth testing and a handful of inputs you already know the right answer to. Real examples from production logs beat invented ones.

Run Your First Eval in Five Steps

Finish these five steps and you have a working, repeatable eval. Everything after this section is depth you can add later.

Scaffold a project. The interactive walkthrough writes a starter config into a new folder.
```
npx promptfoo@latest init my-eval
cd my-eval
```
Prefer a ready-made sample? Run npx promptfoo@latest init --example getting-started instead.
Export your API key in the same shell. Promptfoo reads it from the environment, never from the config file.
```
export OPENAI_API_KEY=sk-your-key-here
```

Write the eval. Replace the generated promptfooconfig.yaml with your own prompt, the models to compare, and one test case per input. Each test lists the assertions that must pass.

prompts:
  - "Summarize this support ticket in one sentence, then classify it as billing, bug, or feature: {{ticket}}"

providers:
  - openai:gpt-4o-mini
  - openai:gpt-4o
  - anthropic:messages:claude-opus-4-8

defaultTest:
  options:
    provider: openai:gpt-4o   # grader for any model-graded assertion

tests:
  - vars:
      ticket: "I was charged twice for my May invoice and need a refund."
    assert:
      - type: icontains
        value: billing
      - type: latency
        threshold: 4000
  - vars:
      ticket: "The export button throws a 500 error every time I click it."
    assert:
      - type: icontains
        value: bug
      - type: llm-rubric
        value: The summary is a single sentence and names the export failure.

Run the eval. Promptfoo runs every prompt against every provider and every test, then prints a pass/fail grid in the terminal.
```
npx promptfoo@latest eval
```
Open the web view to compare outputs cell by cell and see exactly which assertion failed and why.
```
npx promptfoo@latest view
```

That is the whole loop. Change a prompt, rerun eval, and the grid tells you if you improved things or broke them.

Assertions Decide What Passing Means

An eval is only as honest as its assertions. Weak checks pass everything, which is worse than no eval because it buys false confidence. Promptfoo splits assertions into two families, and good suites use both.

Deterministic Checks Catch the Obvious Failures

These run in plain code, cost nothing, and never flake. Reach for them first.

Assertion	What it checks	Use it for
`contains` / `icontains`	Output includes a string (case-insensitive variant)	Required labels, keywords, entity names
`equals`	Output matches exactly	Fixed classifications, canned replies
`regex`	Output matches a pattern	Formats like dates, IDs, phone numbers
`is-json`	Output parses as valid JSON, optionally against a schema	Structured outputs and tool arguments
`cost`	Spend per call stays under a threshold	Catching a model swap that triples cost
`latency`	Response time stays under a threshold in milliseconds	Guarding a user-facing path
`javascript` / `python`	Your own function returns pass/fail	Any rule you can express in code

A threshold assertion reads exactly as you would expect:

assert:
  - type: latency
    threshold: 2000
  - type: cost
    threshold: 0.01

Model-Graded Checks Judge the Fuzzy Stuff

Some qualities resist a regex: whether the tone is right, whether a summary stays faithful, whether the model refused when it should have. For those, llm-rubric hands the output to a grading model with a plain-language standard.

assert:
  - type: llm-rubric
    value: The reply is polite, admits the refund, and gives no legal advice.

The grader defaults to an OpenAI model. Point it at a model your key can call by setting provider once under defaultTest.options, as the step-3 config does, or per assertion. Other model-graded types cover common needs: factuality checks output against facts you supply, answer-relevance checks that the answer addresses the question, and context-faithfulness checks that a retrieval-augmented answer stays inside its retrieved context.

Keep model-graded checks for what deterministic checks cannot express. They cost tokens, add latency, and introduce a little grading variance, so leaning on them for anything a contains would catch is wasteful.

Wire the Eval Into CI So Regressions Fail the Build

An eval you have to remember to run is an eval you will forget. Put it on every pull request that touches a prompt. Promptfoo returns a non-zero exit code when assertions fail, which fails the job on its own. Write results to JSON when you want a custom gate, for example tolerating a small number of failures.

name: LLM Eval
on:
  pull_request:
    paths:
      - "prompts/**"
      - "promptfooconfig.yaml"
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
      - name: Run eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json
      - name: Quality gate
        run: |
          FAILURES=$(jq '.results.stats.failures' results.json)
          echo "Failures: $FAILURES"
          if [ "$FAILURES" -gt 0 ]; then exit 1; fi

Store the provider key as a repository secret, never in the config. Add --share to the eval command when you want a hosted link to the run attached to the pull request.

Fix the Errors That Trip Up First Runs

The failures below account for most stuck first attempts.

A 401 or "invalid API key." The key is missing from the current shell. Re-export it in the same terminal, and confirm the variable name matches the provider (OPENAI_API_KEY, ANTHROPIC_API_KEY).
A model-graded assertion errors out or calls a model you lack access to. The default grader is an OpenAI model. Set defaultTest.options.provider to a model your key can reach.
Rate-limit or 429 errors on larger suites. Lower concurrency with -j 1 (or a small number) so promptfoo fires fewer requests at once.
Stale results after editing the config. Promptfoo caches provider responses. Force fresh calls with npx promptfoo@latest eval --no-cache, or wipe the cache with npx promptfoo@latest cache clear.
YAML parse errors. Almost always indentation. Each entry under assert is a list item, with type and value at the same level beneath it.

What to Do Next

The first eval is a foundation. These four moves turn it into a real safety net.

Feed it real data. Load test cases from a CSV or your production logs instead of hand-writing each one; the datasets docs show the syntax.
Assert your JSON contracts. If a prompt returns structured data, pair is-json with a schema so a malformed field fails the build. See how to get reliable JSON from an LLM with structured outputs.
Guard spend and speed. Add cost and latency thresholds so a model swap cannot quietly triple your bill, a companion to cutting LLM API costs with prompt caching.
Measure task success, not vibes. Rubrics are a start; for agents, track reliability across many runs, as covered in measuring AI agent reliability beyond pass@1 and closing the enterprise AI agent evaluation gap.