Skip to content
BENIBIX Learn

Eval Set

Also known as Evaluation Set, Evaluation Dataset, Golden Set, Golden Dataset

A fixed list of test questions with good answers. You use it to grade an AI app.

Think of it like

It is like a box of practice questions you keep. You use the same box each time, so scores can be compared.

Example

Jack's eval set started with 50 questions. Each time a user found a wrong answer, he added that question, and a year later it had 400.

Why it matters

Scores can only be compared on the same questions. A fixed eval set makes every test fair.

Definition

An eval set is a fixed list of test inputs, each with what a good answer should be. You use it to grade an AI model or app. Each item might hold a question and the right answer, or a rule the answer must follow. Good eval sets mix common cases with hard and odd ones, often taken from real user questions. The set is kept apart from all the data the model was trained or tuned on. It grows over time, as each new failure becomes a new test.

Example code

json
{"input": "How do I reset my password?", "expected": "mentions the Forgot password link"}
One item from an eval set: a test question and what a good answer must do.
Each item holds an input and what a good answer must do. Every new failure adds an item.

Common mistake

Filling the set only with easy questions. A model that passes them can still fail on the hard cases users bring.

Key takeaways

  1. 1
    A fixed list. The same each time.
  2. 2
    Input and good answer. In each item.
  3. 3
    Grows from failures. Each bug a test.

Compare