# DeepEval

> A free Python tool that tests AI answers like unit tests. Each must pass a score.

**Field:** Artificial Intelligence  
**Category:** Evaluation & Benchmarks  
**Also known as:** DeepEval Framework, deepeval test run

## Think of it like

It is like a spelling test for a robot's answers. Each answer must reach a mark, or the test fails.

## Definition

DeepEval is a free, open-source Python tool for testing AI apps. It works like pytest, a tool for testing normal code: you write test cases in a file, and one command runs them all. Each case holds an input, the app's answer and sometimes the answer you expected. Metrics then score the answer from 0 to 1, and a case fails below the line you set. Most metrics use a judge model, such as the checks for relevance, faithfulness and made-up facts. It also has metrics for agents and chats.

## Example

Emma adds 30 DeepEval tests to her support bot's code. They run each night with her other tests, and one failing faithfulness check stops a bad release.

## Example code

```python
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_refund():
    case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="You have 30 days to get a full refund.",
    )
    assert_test(case, [AnswerRelevancyMetric(threshold=0.7)])
```

One test case. It fails if the answer scores under 0.7 for relevance.

## Why it matters

It puts AI checks in the same place as the code tests. A bad prompt change then fails the build, like a bug would.

## Common mistake

Trusting a score without reading why. The judge model can be wrong, so check a sample by hand.

## Three principles

- **Like pytest.** Test files and one command.
- **Scored metrics.** Each scores 0 to 1 against a line.
- **Judge models.** Most metrics ask another model.

## Related terms

- [LLM Evaluation](https://learn.benibix.com/term/ai/llm-evaluation)
- [LLM-as-a-Judge](https://learn.benibix.com/term/ai/llm-as-a-judge)
- [Promptfoo](https://learn.benibix.com/term/ai/promptfoo)
- [Ragas](https://learn.benibix.com/term/ai/ragas)
- [Faithfulness (AI)](https://learn.benibix.com/term/ai/faithfulness)
- [Eval Set](https://learn.benibix.com/term/ai/eval-set)

## Told apart from

- [DeepEval vs Ragas](https://learn.benibix.com/compare/deepeval-vs-ragas): People think DeepEval and Ragas do the same job, and they mostly do not. Ragas is built to score apps that answer from your own documents. It checks the search and the answer apart. DeepEval is a test runner that works like pytest. It runs many kinds of metric and fails the build when a score drops. DeepEval even lists a Ragas score among its metrics.

![Three test cases as bars on a scale from 0 to 1, with a dashed line at 0.7. Refund rules scores 0.91 and passes. Shipping time scores 0.84 and passes. Lost parcel, highlighted, scores 0.52, under the line, and fails. A note says one command runs every case, like a normal test suite.](https://learn.benibix.com/diagrams/ai-deepeval.svg)

---

Source: https://learn.benibix.com/term/ai/deepeval
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
