Terminal-Bench: putting AI agents to work in the terminal
Terminal-Bench is a benchmark that hands an AI agent a concrete job to complete on its own inside a terminal, then checks automatically whether the result is correct. It is used to compare coding agents, and the models behind them, on work close to what a developer or sysadmin actually does.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|---|
| Origin | 2025 |
| Introduced by | Merrill, Shaw et al. (Stanford, Laude Institute) |
What Terminal-Bench measures
A coding AI agent does more than write text: it types commands, reads error messages, edits files and tries again. Terminal-Bench checks whether that loop actually gets the job done. Every task ships with a plain-language instruction, its own Docker environment, a human-written and verified reference solution, and a test script that inspects the final state of the container. The agent only sees the instruction and a shell; the grade depends solely on whether the tests pass, not on how convincing its explanations sound.
Tasks are inspired by real workflows: running a scientific computation, configuring a network, analysing data, calling APIs, playing a game or patching a security vulnerability. The headline metric is a resolution rate, usually reported alongside cost and token usage.
Origins and versions
Terminal-Bench launched on 19 May 2025, built by a team from Stanford and the Laude Institute (including Mike Merrill, Alex Shaw, Ludwig Schmidt and Andy Konwinski). Version 2.0, released in November 2025, went through much stricter verification after the community spotted flaky tasks, and came with a new evaluation harness called Harbor. It is described in the paper “Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces” (January 2026). Further revisions followed in 2026, notably removing “saturated” tasks that every recent model solves reliably.
Why it matters, and where it falls short
Terminal-Bench is one of the few benchmarks that evaluate an agent end to end, model and scaffolding together. That is why it shows up in so many launch announcements for models aimed at agentic coding.
Read the numbers with a few caveats in mind:
- A score describes a model plus an agent. The same model can land very different results depending on the harness wrapped around it.
- Versions are not interchangeable. A figure without a version number tells you little.
- Contamination is a real risk. Tasks are public; the maintainers embed a canary string so they can be excluded from training data, and they retire tasks whose solutions circulate online.
- A test bench is not production. Containers are sandboxed, so an error has no consequences there.
Our AI ranking brings together dated, public benchmark scores, and our methodology explains where they come from and how to read them.
Related terms
- Benchmark · A standardised test, a fixed set of scored questions or tasks, used to compare how different AI models perform.
- Agent evaluation · Methods for measuring whether an AI agent really completes a task end to end, reliably, safely and at an acceptable cost.
- SWE-bench · A benchmark that asks an AI to resolve real GitHub issues from Python projects, with each fix checked against the project's own tests.
- Agentic coding · Software development in which an AI agent reads the codebase, edits files, runs tests and fixes its own mistakes over several steps, under supervision.
- Sandboxing · Running an AI agent inside an isolated environment with restricted files, network and commands, so any damage stays contained.
- AI agent · An AI that does more than answer: it plans and carries out a series of actions with tools to reach a goal you set.
Full glossary
Frequently asked questions
How is Terminal-Bench different from SWE-bench?
SWE-bench asks an agent to fix real bugs in open-source GitHub repositories, and judges the patch with the project's own tests. Terminal-Bench covers a wider range of terminal work: networking, data, security, scientific computing, even games. The two complement each other.
Why are there several versions?
The strongest agents eventually solve some tasks every single time, so those tasks no longer separate one model from another. The maintainers publish revisions that fix broken tasks, drop saturated ones and add harder ones. A score only means something alongside its version number.
Does a high score mean I can let an agent loose on my server?
No. The benchmark runs in isolated, disposable containers where mistakes cost nothing. On a real machine, keep the agent sandboxed, give it minimal permissions and require human approval for sensitive actions.
Who runs Terminal-Bench?
It is led by researchers from Stanford and the Laude Institute, with a large community of contributors who submit tasks. The task set and the evaluation harness are public on the official website.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.