Skip to content
BENIBIX Learn

Promptfoo

Also known as Promptfoo CLI, promptfoo eval, promptfooconfig.yaml

A free tool that tests your prompts on many AI models at once.

Think of it like

It is like a spelling test for a robot. You can give it again and again.

Example

Lucy changes her support bot's prompt. She runs promptfoo eval, and 3 of her 40 test cases now fail, so she fixes them before anyone sees.

Why it matters

A prompt change that fixes one answer can break five others. Running the same tests every time catches that.

Definition

Promptfoo is a free, open-source tool for testing prompts and AI apps. You write a short file that lists your prompts, the models to try, and test cases with the answers you expect. One command runs every test on every model and shows a table of passes and fails. It works with models from many labs, such as OpenAI and Anthropic. It can also attack your app on purpose, to find weak spots like prompt injection. In March 2026, OpenAI agreed to buy it, and said the tool would stay free and open.

Example code

yaml
prompts:
  - 'Answer in one word. What is the capital of {{country}}?'
providers:
  - anthropic:messages:claude-haiku-5-5
tests:
  - vars:
      country: France
    assert:
      - type: icontains
        value: Paris
One prompt, one model and one test. It passes only if the answer says Paris.
Every test runs on every model. One failing cell shows where the prompt broke.

Common mistake

Testing a prompt by trying it once or twice by hand. A few lucky answers prove little, and a fixed set of cases shows the real rate.

Key takeaways

  1. 1
    Tests in a file. Prompts, models and cases in one YAML file.
  2. 2
    Every model at once. Compare answers side by side.
  3. 3
    Red teaming too. It attacks your app on purpose.