FRENES

Glossary · Measuring and comparing

SimpleQA

SimpleQA is a benchmark for checking how accurately a language model answers short factual questions. It compares responses with reference answers to assess factual correctness, while covering only part of the reliability you may need from an assistant when researching information, working with documents, or producing longer material.

SimpleQA: the explanation

The basic approach is to ask a question with a specific factual answer and compare the model’s response with a reference. This creates a narrower grading problem than reviewing a long passage containing many claims. Ambiguity can still arise, so the marking rules matter. You need to know how incorrect answers and decisions not to answer are handled. The [SimpleQA introduction](https://openai.com/index/introducing-simpleqa/) describes this focus on short factual responses and explains why the evaluation uses that limited scope.

For your own work, SimpleQA can provide one signal when selecting an assistant for information requests. You should define your requirements separately from the benchmark. Drafting an exhibition caption, summarizing a case file, and checking recent news are different activities. Add questions that reflect your documents and audience rather than assuming a general result transfers directly. The [SimpleQA research paper](https://cdn.openai.com/papers/simpleqa.pdf) gives the original methodology, which you can examine before deciding how much relevance it has to your intended use.

A reported score needs its testing conditions. Ask whether the model relied on its trained knowledge or had access to additional tools. Also check the exact question collection and grading procedure. For a practical decision, investigate mistakes that would create extra work for you and consider how the system communicates uncertainty. An assistant may be useful in a familiar subject area while requiring closer checks elsewhere. A comparison becomes more informative when the evaluation resembles the information requests you actually make.

A concrete example

You are preparing information cards for a community exhibition. Your assistant must identify places, authors, and events in response to short questions. You review its SimpleQA results as an initial signal, then try questions from your own material. Before publishing a card, you verify its key facts against appropriate sources. You also check whether the assistant acknowledges uncertainty instead of supplying a confident guess when the information is difficult to establish.

Why it matters

SimpleQA highlights the difference between fluent writing and factual correctness. When selecting an assistant, consider what happens on difficult questions as well as on easy ones. Review errors, abstentions, and testing conditions. If a wrong fact would have meaningful consequences for your work, arrange checks for important claims. A benchmark score summarizes past responses to a question set and cannot guarantee the correctness of the next answer you receive.

Common misconceptions

  • The name SimpleQA does not mean that its questions are necessarily easy for a model, because a short question with a short expected answer can still require knowledge that the system cannot retrieve accurately or confidently.
  • A strong result does not guarantee that every statement in a long report will be correct, because answering a brief factual question and writing a document containing many connected claims place different demands on an assistant.

Related terms: measuring and comparing

benchmark
Benchmark is a structured test with defined rules for comparing system performance.
Epoch Capabilities Index
Epoch Capabilities Index is a composite measure that combines results from multiple benchmarks to compare AI models.
SWE-bench
SWE-bench is a benchmark for resolving software issues drawn from real projects.
GPQA Diamond
GPQA Diamond is a selected subset of GPQA containing difficult multiple-choice science questions.
FrontierMath
FrontierMath is an advanced mathematics benchmark developed by Epoch AI with mathematical specialists.
OSWorld
OSWorld is a family of benchmarks and environments for evaluating agents that perform computer tasks.
Terminal-Bench
Terminal-Bench is a collection of tests for measuring how well an AI agent completes tasks through a terminal, an interface controlled with text commands.
AIME (benchmark)
AIME (benchmark) refers to using problems from the American Invitational Mathematics Examination to evaluate language models.

Where this site covers it

The term appears on 91 pages of this site. A few of them:

Go further

Frequently asked questions

What counts as a correct answer?

The response must supply the expected information under the evaluation’s grading rules. Polished wording alone is insufficient. Check how the protocol handles partial answers, extra claims, and alternative wording before deciding what a reported result actually tells you.

Can the model search the web?

That depends on the protocol behind the result you are reading. Tests with browsing and tests without tools measure different configurations. Confirm what access was allowed before comparing scores or applying the result to your own assistant.

Is the score enough?

No. Add trials based on your subject area and inspect incorrect responses. An overall score summarizes performance across a question collection, but it cannot establish whether your next request will be understood or answered correctly.

Neutrality. No affiliate links. No vendor pays to be reviewed, rated or ranked. Reviews are directed by François Chollet, neutral and factual, and every data point is dated and sourced. How we score