Skip to content
BENIBIX Learn

Evaluation & Benchmarks

Measuring how good a model really is · 40 terms

Sort
Compare

Evaluation Basics

4

The core ideas behind testing a model. Right answers, a score to beat, and fair splits of the data.

Checking how good a model is. You test it on data it has not seen.

The right answers a model is checked against. People often make them by hand.

A very simple model you build first. Any real model has to beat its score.

You test a model many times, each on a new slice of the data. Then you take the average.

Classification Metrics

7

The scores used for yes or no models, and the table they are all worked out from.

The share of a model's answers that are right. Nine right out of ten is 90%.

When a model says yes, how often it is right. Few false alarms means high precision.

Out of all the real yes cases, how many a model finds. Few misses means high recall.

One score that blends precision and recall. It stays low if either one is low.

A table of a model's right and wrong answers. It shows what kind of mistakes it makes.

A curve that shows catches against false alarms. AUC sums it up in one number.

The cut-off that turns a model's score into yes or no. Moving it trades misses for false alarms.

Regression Metrics

3

Scores for models that guess a number, such as a price or a temperature, and how far off they are.

The average size of a model's misses when it guesses numbers. Lower is better.

An error score for number guesses that punishes big misses more. Lower is better.

A score for models that guess numbers. 1 is perfect, and 0 is no better than the average.

Text Generation Metrics

3

Automatic scores for text a model writes, from how well it predicts words to how much it matches a person's version.

A score for how well a language model predicts text. Lower means less surprised.

A score for machine translation. It counts word runs that match a person's translation.

A set of scores for summaries. They check how much of a person's summary was covered.

LLM Testing

10

How free text answers from AI models are graded. Saved test sets, rubrics, other models and people all do it.

Tests that check if an AI gives good answers. Code, a model or people grade them.

A fixed list of test questions with good answers. You use it to grade an AI app.

One AI model grades another's answers. It is fast and cheap, but not perfect.

People judge the answers of an AI model. It is slow, but closest to what users think.

A written checklist for grading AI answers. Each line says what earns a point.

Two AI answers are put side by side. The grader picks the better one.

Whether an AI answer only says what its source text backs up. It matters most when the AI answers from your files.

A free tool that tests your prompts on many AI models at once.

A free Python tool that tests AI answers like unit tests. Each must pass a score.

A free Python tool. It scores AI apps that answer from your own documents.

Benchmarks and Leaderboards

13

Public tests that rank AI models, the best known of them, and how their scores get spoiled.

A public test that many AI models take. Their scores can then be compared.

A test that hides one fact in a long text and asks the model to find it.

A public table that ranks AI models by test scores or votes. The order changes often.

A site where people vote between two hidden AI models. The votes become a ranking.

When a model saw the test questions while it learned. Its score is then too high.

Top models all score near full marks. So the test can no longer tell them apart.

Knowledge and Reasoning Benchmarks

4

Public tests of what a model knows and how well it reasons, from school subjects to expert science and grid puzzles.

A test of about 16,000 questions over 57 subjects. Each has four choices.

A test of 448 hard science questions. A web search does not easily answer them.

A test of 2,500 expert questions over 100+ subjects. It was built to stay hard.

Grid puzzles. Each one asks an AI to learn a new rule from just a few pairs.

Code Benchmarks

3

Public tests that check AI code by running it, and the pass@k score they report.

164 Python tasks, each checked by running unit tests. A classic coding test for AI.

The share of coding tasks a model solves within k tries. Pass@1 means one try.

A test where AI fixes real bugs in real code. The project's tests check each fix.