SWE-bench: can an AI fix real-world bugs?
SWE-bench gives an AI a real GitHub issue along with the project's code, then checks whether its fix makes the project's tests pass. It has become the most widely cited yardstick for comparing coding agents.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|---|
| Origin | 2023 |
| Introduced by | Jimenez et al. |
Where SWE-bench comes from
The benchmark was introduced in “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (Carlos E. Jimenez, John Yang et al., Princeton, October 2023, published at ICLR 2024). The authors collected 2,294 problems drawn from real GitHub issues and their matching pull requests, across 12 popular Python repositories. Their starting point: existing coding benchmarks, made of small self-contained exercises, no longer reflected how hard real work is.
What it measures
For each task, the system receives the repository as it stood when the issue was filed, plus the issue description. It must produce a patch. That patch is applied and selected tests are run: tests that failed before the real fix must now pass, and tests that already passed must keep passing. Solving a task therefore means navigating a whole project, understanding existing code and often editing several files, which is exactly what an agentic coding tool does.
Several variants now exist: Lite (smaller and cheaper to run), Verified (human-reviewed), Multilingual (beyond Python) and Multimodal (issues that include images).
Its limits
- A narrow scope: the original version only covers Python and well-known open-source libraries. Results say nothing about, say, a company’s internal app or another language.
- Contamination: the issues and their fixes have been public for a long time and may have been seen during model training.
- Imperfect tests: a patch can pass the tests while being brittle, unreadable or incomplete. Code quality is not scored.
- Setup dependence: the score measures a model plus a given scaffold, not a model on its own.
Why it matters
SWE-bench moved agent evaluation from isolated questions to end-to-end tasks that can be checked automatically. It remains a good trend indicator, best read alongside other measures such as Terminal-Bench. Current scores appear in the AI ranking, and our methodology explains where they come from.
A common myth
- “A high score means the AI codes like an engineer.” It means the AI handles this kind of issue well, under these test conditions.
Related terms
- Benchmark · A standardised test, a fixed set of scored questions or tasks, used to compare how different AI models perform.
- Agentic coding · Software development in which an AI agent reads the codebase, edits files, runs tests and fixes its own mistakes over several steps, under supervision.
- Agent evaluation · Methods for measuring whether an AI agent really completes a task end to end, reliably, safely and at an acceptable cost.
- Terminal-Bench · A benchmark that measures whether an AI agent can complete hard, realistic tasks on its own inside a command-line terminal.
- GPQA · A benchmark of PhD-level multiple-choice questions in biology, physics and chemistry, designed so a web search does not reveal the answer.
Try it
Full glossary
Frequently asked questions
What does a SWE-bench score mean?
It is the share of issues for which the proposed fix makes the designated tests pass. A high score means a system often resolves this kind of bug in these projects, not that it can replace a developer on any codebase.
What is the difference between SWE-bench and SWE-bench Verified?
Verified is a 500-task subset of the original test set, reviewed by human annotators to weed out poorly described issues, incorrect tests and unsolvable tasks. It was built with OpenAI and is the variant most often quoted.
Does the score depend only on the model?
No. It also depends on the scaffolding around the model: the tools provided, the number of attempts, the time allowed, how it searches the code. Two scores obtained with different setups cannot be compared directly.
Why doesn't the site show every score vendors announce?
The site relies on public sources with documented test conditions. Figures announced by vendors themselves often use their own setups, which makes them hard to compare.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.