# Promptfoo

> A free tool that tests your prompts on many AI models at once.

**Field:** Artificial Intelligence  
**Category:** Evaluation & Benchmarks  
**Also known as:** Promptfoo CLI, promptfoo eval, promptfooconfig.yaml

## Think of it like

It is like a spelling test for a robot. You can give it again and again.

## Definition

Promptfoo is a free, open-source tool for testing prompts and AI apps. You write a short file that lists your prompts, the models to try, and test cases with the answers you expect. One command runs every test on every model and shows a table of passes and fails. It works with models from many labs, such as OpenAI and Anthropic. It can also attack your app on purpose, to find weak spots like prompt injection. In March 2026, OpenAI agreed to buy it, and said the tool would stay free and open.

## Example

Lucy changes her support bot's prompt. She runs promptfoo eval, and 3 of her 40 test cases now fail, so she fixes them before anyone sees.

## Example code

```yaml
prompts:
  - 'Answer in one word. What is the capital of {{country}}?'
providers:
  - anthropic:messages:claude-haiku-5-5
tests:
  - vars:
      country: France
    assert:
      - type: icontains
        value: Paris
```

One prompt, one model and one test. It passes only if the answer says Paris.

## Why it matters

A prompt change that fixes one answer can break five others. Running the same tests every time catches that.

## Common mistake

Testing a prompt by trying it once or twice by hand. A few lucky answers prove little, and a fixed set of cases shows the real rate.

## Three principles

- **Tests in a file.** Prompts, models and cases in one YAML file.
- **Every model at once.** Compare answers side by side.
- **Red teaming too.** It attacks your app on purpose.

## Related terms

- [LLM Evaluation](https://learn.benibix.com/term/ai/llm-evaluation)
- [Eval Set](https://learn.benibix.com/term/ai/eval-set)
- [LLM-as-a-Judge](https://learn.benibix.com/term/ai/llm-as-a-judge)
- [AI Red Teaming](https://learn.benibix.com/term/ai/ai-red-teaming)
- [DeepEval](https://learn.benibix.com/term/ai/deepeval)
- [Ragas](https://learn.benibix.com/term/ai/ragas)
- [YAML Ain't Markup Language (YAML)](https://learn.benibix.com/term/development/yaml)

![A file called promptfooconfig.yaml, listing prompts, providers and tests, feeds an eval arrow into a grid. Three tests run on model A and model B. Every cell says pass except test 2 on model B, which is highlighted and says fail.](https://learn.benibix.com/diagrams/ev-promptfoo.svg)

---

Source: https://learn.benibix.com/term/ai/promptfoo
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
