Skip to content
BENIBIX Learn

SWE-bench

AI📏 Evaluation & BenchmarksBenchmarks and Leaderboards

Also known as SWE-bench Verified, Software Engineering Benchmark

A test where AI fixes real bugs in real code. The project's tests check each fix.

Think of it like

It is like giving a car fixer a real broken car, not a quiz about cars. Then you check if it runs.

Example

An AI coding tool claims 70% on SWE-bench Verified. That means its fixes passed the tests on about 350 of the 500 tasks.

Why it matters

It tests real work, not quiz answers. That makes it a good guide to how well a coding agent might help.

Definition

SWE-bench is a benchmark that tests if AI can fix real bugs in real code. Each task is a real issue from one of 12 popular Python projects on GitHub, with the code as it was at that time. The model must change the code, and the project's own tests check whether the fix works. The full set has 2,294 tasks. SWE-bench Verified is a set of 500 of them that people checked by hand, and it is the version most often quoted.
A real issue goes in, the model changes the code, and the project's tests decide.

Common mistake

Assuming a high score means it can fix bugs in any code. All the tasks are in Python, from a small set of well-known projects.

Key takeaways

  1. 1
    Real bugs. From real projects.
  2. 2
    Tests decide. Not a person.
  3. 3
    Verified set. 500 checked tasks.

Compare