📏SWE-bench
Also known as SWE-bench Verified, Software Engineering Benchmark
A test where AI fixes real bugs in real code. The project's tests check each fix.
Think of it like
It is like giving a car fixer a real broken car, not a quiz about cars. Then you check if it runs.
Example
An AI coding tool claims 70% on SWE-bench Verified. That means its fixes passed the tests on about 350 of the 500 tasks.
Why it matters
It tests real work, not quiz answers. That makes it a good guide to how well a coding agent might help.
Definition
Common mistake
Key takeaways
- 1Real bugs. From real projects.
- 2Tests decide. Not a person.
- 3Verified set. 500 checked tasks.