# Eval Set

> A fixed list of test questions with good answers. You use it to grade an AI app.

**Field:** Artificial Intelligence  
**Category:** Evaluation & Benchmarks  
**Also known as:** Evaluation Set, Evaluation Dataset, Golden Set, Golden Dataset

## Think of it like

It is like a box of practice questions you keep. You use the same box each time, so scores can be compared.

## Definition

An eval set is a fixed list of test inputs, each with what a good answer should be. You use it to grade an AI model or app. Each item might hold a question and the right answer, or a rule the answer must follow. Good eval sets mix common cases with hard and odd ones, often taken from real user questions. The set is kept apart from all the data the model was trained or tuned on. It grows over time, as each new failure becomes a new test.

## Example

Jack's eval set started with 50 questions. Each time a user found a wrong answer, he added that question, and a year later it had 400.

## Example code

```json
{"input": "How do I reset my password?", "expected": "mentions the Forgot password link"}
```

One item from an eval set: a test question and what a good answer must do.

## Why it matters

Scores can only be compared on the same questions. A fixed eval set makes every test fair.

## Common mistake

Filling the set only with easy questions. A model that passes them can still fail on the hard cases users bring.

## Three principles

- **A fixed list.** The same each time.
- **Input and good answer.** In each item.
- **Grows from failures.** Each bug a test.

## Related terms

- [LLM Evaluation](https://learn.benibix.com/term/ai/llm-evaluation)
- [Test Set](https://learn.benibix.com/term/ai/test-set)
- [Ground Truth](https://learn.benibix.com/term/ai/ground-truth)
- [Benchmark Contamination](https://learn.benibix.com/term/ai/benchmark-contamination)

## Told apart from

- [AI Benchmark vs Eval Set](https://learn.benibix.com/compare/ai-benchmark-vs-eval-set): People often think a high benchmark score means a model will do well in their own app. It may not, since the two tests ask different things. A benchmark is a public test, the same for every model, so it measures general skill. An eval set is your own list of test questions, built from your users and your task. Use benchmarks to choose which models to try, and your eval set to decide.

![A list of test items, each with an input and what a good answer must do. New items join the list each time a user finds a wrong answer.](https://learn.benibix.com/diagrams/ev-eval-set.svg)

---

Source: https://learn.benibix.com/term/ai/eval-set
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
