📏HumanEval
Also known as HumanEval Benchmark, OpenAI HumanEval
164 Python tasks, each checked by running unit tests. A classic coding test for AI.
Think of it like
It is like a set of small homework tasks for coders. Each answer is checked by running it.
Example
Emma runs HumanEval on a small open model. It solves 98 of the 164 tasks on the first try, so its pass@1 is about 60%.
Why it matters
It checks code by running it, not by reading it. Code that looks right but fails a test gets no credit.
Definition
Example code
python
def add_up(numbers):
"""Return the sum of the list."""
# test
assert add_up([1, 2, 3]) == 6Common mistake
Key takeaways
- 1Run, do not read. Tests decide pass or fail.
- 2Small tasks. One function each.
- 3Python only. 164 tasks in one language.