Skip to content
BENIBIX Learn

DeepEval

Also known as DeepEval Framework, deepeval test run

A free Python tool that tests AI answers like unit tests. Each must pass a score.

Think of it like

It is like a spelling test for a robot's answers. Each answer must reach a mark, or the test fails.

Example

Emma adds 30 DeepEval tests to her support bot's code. They run each night with her other tests, and one failing faithfulness check stops a bad release.

Why it matters

It puts AI checks in the same place as the code tests. A bad prompt change then fails the build, like a bug would.

Definition

DeepEval is a free, open-source Python tool for testing AI apps. It works like pytest, a tool for testing normal code: you write test cases in a file, and one command runs them all. Each case holds an input, the app's answer and sometimes the answer you expected. Metrics then score the answer from 0 to 1, and a case fails below the line you set. Most metrics use a judge model, such as the checks for relevance, faithfulness and made-up facts. It also has metrics for agents and chats.

Example code

python
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_refund():
    case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="You have 30 days to get a full refund.",
    )
    assert_test(case, [AnswerRelevancyMetric(threshold=0.7)])
One test case. It fails if the answer scores under 0.7 for relevance.
Each test case gets a score. Under the line, the test fails.

Common mistake

Trusting a score without reading why. The judge model can be wrong, so check a sample by hand.

Key takeaways

  1. 1
    Like pytest. Test files and one command.
  2. 2
    Scored metrics. Each scores 0 to 1 against a line.
  3. 3
    Judge models. Most metrics ask another model.

Compare