HumanEval vs SWE-bench
People quote HumanEval and SWE-bench as if both meant the same thing: can it code? HumanEval has 164 short tasks. Each is one function, checked by tests written for it. SWE-bench gives a model real bug reports from large Python projects, and each project's own tests decide. A model can score high on HumanEval and still struggle on SWE-bench. Finding the right files in a big codebase is much of the work.
HumanEval
164 Python tasks, each checked by running unit tests. A classic coding test for AI.
HumanEval is a benchmark of 164 small Python tasks, released by OpenAI in 2021. Each task gives a function name and a short note on what it should do. The model writes the code. The code is then run against unit tests written by hand, about 7.7 per task. A task counts as solved only if every test passes. The tasks are short, so bigger tests like SWE-bench are used for real projects.
SWE-bench
A test where AI fixes real bugs in real code. The project's tests check each fix.
SWE-bench is a benchmark that tests if AI can fix real bugs in real code. Each task is a real issue from one of 12 popular Python projects on GitHub, with the code as it was at that time. The model must change the code, and the project's own tests check whether the fix works. The full set has 2,294 tasks. SWE-bench Verified is a set of 500 of them that people checked by hand, and it is the version most often quoted.
In one line
- HumanEval
- 164 Python tasks, each checked by running unit tests. A classic coding test for AI.
- SWE-bench
- A test where AI fixes real bugs in real code. The project's tests check each fix.