GPQA: expert questions that Google alone can't crack

Origin and purpose

GPQA was introduced in “GPQA: A Graduate-Level Google-Proof Q&A Benchmark” (David Rein, Samuel R. Bowman et al., November 2023). The dataset holds 448 multiple-choice questions written by experts in biology, physics and chemistry, then checked by other experts and tried out on non-specialists.

The authors were after more than a model leaderboard. They wanted a testbed for “scalable oversight”: how humans can keep checking AI answers on topics where the AI knows more than they do. That requires questions only genuine experts can settle, hence the PhD level and the resistance to web searches.

What the benchmark measures

GPQA tests whether a model can combine specialist scientific knowledge with multi-step reasoning, without leaning on an answer that sits ready-made online. That is why it became a reference point for reasoning models and for gauging the effect of test-time compute: letting a model think longer often makes a clear difference here.

Its limits

• Multiple choice: picking from four answers is easier than producing an open answer, and random guessing already earns a non-zero score.
• Small size: with a few hundred questions, and fewer still in Diamond, small gaps between models are often statistical noise.
• Saturation: once top models get close to the ceiling, the benchmark no longer separates them well.
• Contamination: despite the authors’ precautions, questions may have leaked.
• A narrow field: three sciences, no actions, no tool use, unlike SWE-bench or other forms of agent evaluation.

How to read the scores

A GPQA score signals a trend on one specific kind of intellectual effort, best read alongside other benchmarks. Current scores are in the AI ranking, and our methodology sets out their sources and dates.

A common myth

• “GPQA measures general intelligence.” It measures success on a hard science quiz, which is useful but much narrower.