Skip to content
BENIBIX Learn

Pass@k

AI📏 Evaluation & BenchmarksBenchmarks and Leaderboards

Also known as Pass at k, pass@1, pass@10

The share of coding tasks a model solves within k tries. Pass@1 means one try.

Think of it like

It is like getting three shots at a basketball hoop. You win if any one goes in.

Example

A model solves 60% of tasks with one try and 85% with ten. So its pass@1 is 60% and its pass@10 is 85%.

Why it matters

It shows how often you can trust the first answer, and how much trying again helps.

Formula

pass@k = share of tasks where at least one of k tries passes every test

Definition

Pass@k is a score for code models. The model gets k tries at each task, and the task counts as solved if any try passes all the tests. Pass@1 means one try, which is closest to real use. Pass@10 gives ten tries, so its score is never lower. Benchmarks such as HumanEval report it as the share of tasks solved. To get a steady number, many tries are made and the chance is worked out from them.
With one try this task fails. With three tries it counts as solved, so pass@3 beats pass@1.

Common mistake

Comparing a pass@10 score with a pass@1 score. More tries always look better.

Key takeaways

  1. 1
    k tries. Any one pass counts.
  2. 2
    Pass@1 is strict. One shot, like real use.
  3. 3
    Same k only. Compare like with like.