Skip to content
BENIBIX Learn

Graduate-Level Google-Proof Q&A (GPQA)

AI📏 Evaluation & BenchmarksBenchmarks and Leaderboards

Also known as GPQA, GPQA Diamond, GPQA Benchmark

A test of 448 hard science questions. A web search does not easily answer them.

Think of it like

It is like a quiz so hard that looking things up does not help much.

Example

A model's release notes list its GPQA Diamond score. Jack reads it as a sign of deep science knowledge, not as proof it can run his lab.

Why it matters

Easy quizzes stopped telling top models apart. GPQA asks questions where only real experts do well.

Definition

GPQA is a benchmark of 448 multiple-choice questions in biology, physics and chemistry. Experts with PhDs in those fields wrote them. They are called Google-proof because a web search does not easily give the answer. In the 2023 paper, experts in the field got 65% right. Skilled people from other fields got only 34%, even with the web and over 30 minutes per question on average. A harder subset is called GPQA Diamond.
GPQA is one of the harder tests, with room left at the top that older quizzes lost.

Common mistake

Comparing scores from different sets. A score on GPQA Diamond and a score on the full set are not the same test.

Key takeaways

  1. 1
    Written by experts. PhDs wrote every question.
  2. 2
    Google-proof. Search does not give it away.
  3. 3
    Three sciences. Biology, physics and chemistry.