Skip to content
BENIBIX Learn

HumanEval

AI📏 Evaluation & BenchmarksBenchmarks and Leaderboards

Also known as HumanEval Benchmark, OpenAI HumanEval

164 Python tasks, each checked by running unit tests. A classic coding test for AI.

Think of it like

It is like a set of small homework tasks for coders. Each answer is checked by running it.

Example

Emma runs HumanEval on a small open model. It solves 98 of the 164 tasks on the first try, so its pass@1 is about 60%.

Why it matters

It checks code by running it, not by reading it. Code that looks right but fails a test gets no credit.

Definition

HumanEval is a benchmark of 164 small Python tasks, released by OpenAI in 2021. Each task gives a function name and a short note on what it should do. The model writes the code. The code is then run against unit tests written by hand, about 7.7 per task. A task counts as solved only if every test passes. The tasks are short, so bigger tests like SWE-bench are used for real projects.

Example code

python
def add_up(numbers):
    """Return the sum of the list."""

# test
assert add_up([1, 2, 3]) == 6
The shape of a task: a name and a note to finish, then a test that runs the code.
HumanEval runs every try through the tests. Only a full pass counts.

Common mistake

Treating a high HumanEval score as proof a model can work in a real codebase. Each task is one short function.

Key takeaways

  1. 1
    Run, do not read. Tests decide pass or fail.
  2. 2
    Small tasks. One function each.
  3. 3
    Python only. 164 tasks in one language.

Compare