GPQA Diamond

GPQA Diamond: the explanation

A system receives a scientific question with several possible answers and must select the expected option. Success may require combining specialist knowledge, interpreting assumptions and rejecting plausible alternatives. The reported score reflects final choices under the stated evaluation procedure. A correct choice does not establish that every part of an accompanying explanation is accurate. Conversely, a response can contain some relevant reasoning and still select the wrong answer. Understanding the scoring target helps you distinguish measured accuracy from the quality of the entire response.

GPQA was designed around questions that are not easily answered through a straightforward information search. Diamond refers to a selection informed by the original dataset’s validation process, rather than the complete GPQA collection. The [GPQA research paper](https://arxiv.org/abs/2311.12022) describes the scientific focus and how the questions were constructed. For you, the benchmark provides evidence about particular advanced science capabilities. It does not directly describe how clearly a model explains a topic, follows a document or supports an ordinary conversation.

Multiple-choice questions make scoring relatively straightforward, but they do not reproduce every scientific activity. Selecting an option differs from designing an experiment, checking an unfamiliar result or producing an original discovery. Instructions, computing effort and permitted tools can also affect scores. Public evaluation material may be vulnerable to contamination through training exposure. When comparing results, check the exact subset and evaluation setup. Strong overall performance still leaves individual errors possible, so important claims need verification in the context where you plan to use them.

A concrete example

You want an assistant to help you understand chemistry papers. A strong GPQA Diamond result suggests that a model deserves closer inspection for scientific questions. You then provide an accessible passage and ask it to separate observations, assumptions and conclusions. You check the response against the source document. This practical trial evaluates fidelity and clarity, qualities that success on a multiple-choice benchmark does not establish by itself.

Why it matters

GPQA Diamond can help compare models on demanding science questions, but it remains a specialized indicator. A document assistant also needs to represent sources accurately, explain clearly and acknowledge uncertainty. If an agent uses search or calculation tools, assess those steps separately. A benchmark score does not automatically measure their reliability or guarantee that every response is sound. Match the evidence to the kind of scientific help you actually need.

Common misconceptions

• Strong GPQA Diamond performance does not make a model an autonomous scientist, because answering difficult closed questions does not demonstrate the ability to design experiments, generate discoveries or independently validate unfamiliar results through a complete research process.
• Questions that resist a simple web search are not immune to memorization, because exposure to the questions or solutions during training can still influence scores and complicate claims about a model’s ability to handle genuinely new problems.

Related terms: measuring and comparing

benchmark

Benchmark is a structured test with defined rules for comparing system performance.

Epoch Capabilities Index

Epoch Capabilities Index is a composite measure that combines results from multiple benchmarks to compare AI models.

SWE-bench

SWE-bench is a benchmark for resolving software issues drawn from real projects.

FrontierMath

FrontierMath is an advanced mathematics benchmark developed by Epoch AI with mathematical specialists.

OSWorld

OSWorld is a family of benchmarks and environments for evaluating agents that perform computer tasks.

Terminal-Bench

Terminal-Bench is a collection of tests for measuring how well an AI agent completes tasks through a terminal, an interface controlled with text commands.

SimpleQA

SimpleQA is a benchmark for checking how accurately a language model answers short factual questions.

AIME (benchmark)

AIME (benchmark) refers to using problems from the American Invitational Mathematics Examination to evaluate language models.

Where this site covers it

The term appears on 125 pages of this site. A few of them:

• LLM benchmarks: how to read model rankings Article
• Claude Fable 5: pricing, context and benchmarks Model
• Claude Haiku 4.5: pricing, context and benchmarks Model
• Claude Haiku 5.5: pricing, context and benchmarks Model
• Claude Opus 4.1: pricing, context and benchmarks Model
• Claude Opus 4.5: pricing, context and benchmarks Model
• Claude Opus 4.6: pricing, context and benchmarks Model
• Claude Opus 4.7: pricing, context and benchmarks Model