📏Pass@k
Also known as Pass at k, pass@1, pass@10
The share of coding tasks a model solves within k tries. Pass@1 means one try.
Think of it like
It is like getting three shots at a basketball hoop. You win if any one goes in.
Example
A model solves 60% of tasks with one try and 85% with ten. So its pass@1 is 60% and its pass@10 is 85%.
Why it matters
It shows how often you can trust the first answer, and how much trying again helps.
Formula
pass@k = share of tasks where at least one of k tries passes every testDefinition
Common mistake
Key takeaways
- 1k tries. Any one pass counts.
- 2Pass@1 is strict. One shot, like real use.
- 3Same k only. Compare like with like.