# SWE-bench

> A test where AI fixes real bugs in real code. The project's tests check each fix.

**Field:** Artificial Intelligence  
**Category:** Evaluation & Benchmarks  
**Also known as:** SWE-bench Verified, Software Engineering Benchmark

## Think of it like

It is like giving a car fixer a real broken car, not a quiz about cars. Then you check if it runs.

## Definition

SWE-bench is a benchmark that tests if AI can fix real bugs in real code. Each task is a real issue from one of 12 popular Python projects on GitHub, with the code as it was at that time. The model must change the code, and the project's own tests check whether the fix works. The full set has 2,294 tasks. SWE-bench Verified is a set of 500 of them that people checked by hand, and it is the version most often quoted.

## Example

An AI coding tool claims 70% on SWE-bench Verified. That means its fixes passed the tests on about 350 of the 500 tasks.

## Why it matters

It tests real work, not quiz answers. That makes it a good guide to how well a coding agent might help.

## Common mistake

Assuming a high score means it can fix bugs in any code. All the tasks are in Python, from a small set of well-known projects.

## Three principles

- **Real bugs.** From real projects.
- **Tests decide.** Not a person.
- **Verified set.** 500 checked tasks.

## Related terms

- [AI Benchmark](https://learn.benibix.com/term/ai/ai-benchmark)
- [AI Agent](https://learn.benibix.com/term/ai/ai-agent)
- [Massive Multitask Language Understanding (MMLU)](https://learn.benibix.com/term/ai/mmlu)
- [Unit Test](https://learn.benibix.com/term/development/unit-test)
- [GitHub](https://learn.benibix.com/term/development/github)
- [HumanEval](https://learn.benibix.com/term/ai/humaneval)
- [Pass@k](https://learn.benibix.com/term/ai/pass-at-k)

## Told apart from

- [HumanEval vs SWE-bench](https://learn.benibix.com/compare/humaneval-vs-swe-bench): People quote HumanEval and SWE-bench as if both meant the same thing: can it code? HumanEval has 164 short tasks. Each is one function, checked by tests written for it. SWE-bench gives a model real bug reports from large Python projects, and each project's own tests decide. A model can score high on HumanEval and still struggle on SWE-bench. Finding the right files in a big codebase is much of the work.

![A real issue from a code project goes to the model. The model changes the code, and the project's own tests run on the change. Green means the fix passed.](https://learn.benibix.com/diagrams/ev-swe-bench.svg)

---

Source: https://learn.benibix.com/term/ai/swe-bench
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
