Glossary · Measuring and comparing
OSWorld
OSWorld is a family of benchmarks and environments for evaluating agents that perform computer tasks. Systems interact with applications and files in a controlled setup, and their work is checked to measure actual execution rather than merely their ability to describe how a task could be done.
OSWorld: the explanation
The evaluation prepares an environment containing applications, files and a goal. The agent observes available information and acts through permitted mechanisms, such as mouse and keyboard operations. It must coordinate multiple steps and respond to the effects of its actions. Evaluation mechanisms then inspect the resulting work. This distinguishes an assistant that explains how to edit a document from an agent that actually makes the requested changes in an application. The final result provides evidence beyond a convincing description of what should have happened.
OSWorld studies how agents use computers through real applications. Tasks can involve documents, files and workflows spanning several programs. The family has evolved, with OSWorld 2.0 emphasizing longer assignments and constraints found in more complete workflows. The [official OSWorld 2.0 presentation](https://osworld-v2.xlang.ai/) describes that direction. You therefore need to identify the version behind any score. Results from different environments or task collections do not necessarily measure the same behavior, even when they share the broader OSWorld name in a comparison chart.
Performance depends on the model, observation methods, permitted actions and available step budget. Application configuration can also change task difficulty. Success in a benchmark environment does not guarantee equal reliability on your computer with your accounts and documents. Task completion is not a complete description of safe operation either. When assessing an agent, examine how it handles permissions, missing information, mistakes and actions that are difficult to reverse. These behaviors matter alongside the final success rate when the system works with your actual resources.
A concrete example
You want to delegate preparation of an expense spreadsheet from a folder of receipts. An OSWorld result can provide evidence about an agent’s ability to coordinate actions across applications. You then run a trial with fictional files and a copy of your spreadsheet. You check amounts, changed cells and the location of the saved output. The trial also reveals whether the agent verifies its work or stops before completing every requirement.
Why it matters
OSWorld is relevant when you need an agent that acts in software rather than only discussing a task. Check the benchmark version and the resources used to produce the result. For your situation, also examine completion time, recoverable mistakes and supervision options. Success on some office tasks does not justify unrestricted access. Permissions should match the actions needed, with appropriate review before operations that could have significant consequences for your work.
Common misconceptions
- A strong OSWorld result does not mean an agent will work perfectly in all your applications, because interfaces, configuration, available files and account permissions may differ substantially from those present in the environment used for evaluation.
- Completing an OSWorld task does not establish comprehensive safety, because reaching the expected outcome and respecting every permission, privacy or reversibility requirement are different dimensions that need explicit examination in the setting where the agent will operate.
Related terms: measuring and comparing
- benchmark
- Benchmark is a structured test with defined rules for comparing system performance.
- Epoch Capabilities Index
- Epoch Capabilities Index is a composite measure that combines results from multiple benchmarks to compare AI models.
- SWE-bench
- SWE-bench is a benchmark for resolving software issues drawn from real projects.
- GPQA Diamond
- GPQA Diamond is a selected subset of GPQA containing difficult multiple-choice science questions.
- FrontierMath
- FrontierMath is an advanced mathematics benchmark developed by Epoch AI with mathematical specialists.
- Terminal-Bench
- Terminal-Bench is a collection of tests for measuring how well an AI agent completes tasks through a terminal, an interface controlled with text commands.
- SimpleQA
- SimpleQA is a benchmark for checking how accurately a language model answers short factual questions.
- AIME (benchmark)
- AIME (benchmark) refers to using problems from the American Invitational Mathematics Examination to evaluate language models.
Where this site covers it
The term appears on 9 pages of this site. A few of them:
- Evaluating an AI agent: test sets, metrics, public benchmarks, and production monitoring Guide
- Computer use agents: how they work, available tools, reliability, and security Guide
- LLM benchmarks: how to read model rankings Article
- Claude Opus 4.5: pricing, context and benchmarks Model
- Claude Sonnet 4.5: pricing, context and benchmarks Model
- Claude Sonnet 4.6: pricing, context and benchmarks Model
- Kimi K2.5: pricing, context and benchmarks Model
- o3: pricing, context and benchmarks Model
Go further
Frequently asked questions
Is OSWorld an operating system?
No. It is a project for evaluating agents in computer environments. It uses existing systems and applications to set up tasks and inspect results. It does not replace the operating system already installed on your own computer.
Why does the version matter?
Versions can change tasks, environments and evaluation criteria. OSWorld 2.0 particularly emphasizes longer workflows. A score from an earlier version therefore cannot automatically be treated as a result on the newer collection or used as an equivalent comparison.
Can I let an agent work unsupervised?
A benchmark score is not enough to decide. Start with limited tasks, copies of files and appropriate permissions. Check output quality and require review before actions with significant consequences, especially when mistakes would be difficult to reverse.
Neutrality. No affiliate links. No vendor pays to be reviewed, rated or ranked. Reviews are directed by François Chollet, neutral and factual, and every data point is dated and sourced. How we score