# HumanEval

> 164 Python tasks, each checked by running unit tests. A classic coding test for AI.

**Field:** Artificial Intelligence  
**Category:** Evaluation & Benchmarks  
**Also known as:** HumanEval Benchmark, OpenAI HumanEval

## Think of it like

It is like a set of small homework tasks for coders. Each answer is checked by running it.

## Definition

HumanEval is a benchmark of 164 small Python tasks, released by OpenAI in 2021. Each task gives a function name and a short note on what it should do. The model writes the code. The code is then run against unit tests written by hand, about 7.7 per task. A task counts as solved only if every test passes. The tasks are short, so bigger tests like SWE-bench are used for real projects.

## Example

Emma runs HumanEval on a small open model. It solves 98 of the 164 tasks on the first try, so its pass@1 is about 60%.

## Example code

```python
def add_up(numbers):
    """Return the sum of the list."""

# test
assert add_up([1, 2, 3]) == 6
```

The shape of a task: a name and a note to finish, then a test that runs the code.

## Why it matters

It checks code by running it, not by reading it. Code that looks right but fails a test gets no credit.

## Common mistake

Treating a high HumanEval score as proof a model can work in a real codebase. Each task is one short function.

## Three principles

- **Run, do not read.** Tests decide pass or fail.
- **Small tasks.** One function each.
- **Python only.** 164 tasks in one language.

## Related terms

- [AI Benchmark](https://learn.benibix.com/term/ai/ai-benchmark)
- [Pass@k](https://learn.benibix.com/term/ai/pass-at-k)
- [SWE-bench](https://learn.benibix.com/term/ai/swe-bench)
- [Unit Test](https://learn.benibix.com/term/development/unit-test)
- [Code Generation (AI)](https://learn.benibix.com/term/ai/code-generation)
- [AI Coding Assistant](https://learn.benibix.com/term/ai/ai-coding-assistant)

## Told apart from

- [HumanEval vs SWE-bench](https://learn.benibix.com/compare/humaneval-vs-swe-bench): People quote HumanEval and SWE-bench as if both meant the same thing: can it code? HumanEval has 164 short tasks. Each is one function, checked by tests written for it. SWE-bench gives a model real bug reports from large Python projects, and each project's own tests decide. A model can score high on HumanEval and still struggle on SWE-bench. Finding the right files in a big codebase is much of the work.

![One coding task, and a model's three tries at it. Each try runs through the same unit tests. The first two fail and the third passes.](https://learn.benibix.com/diagrams/ai5-pass-at-k.svg)

---

Source: https://learn.benibix.com/term/ai/humaneval
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
