Skip to content
BENIBIX Learn

LLM Evaluation

Also known as Evals, LLM Evals, AI Evals, AI Evaluation

Tests that check if an AI gives good answers. Code, a model or people grade them.

Think of it like

It is like a driving test on many roads. One good drive does not prove you are a safe driver.

Example

Lucy changes the prompt of her support bot. Her evals run 300 saved questions through it, and the share of good answers drops from 88% to 81%, so she rolls the change back.

Why it matters

Without evals, every prompt change is a guess. With them, you can see if a change helped or hurt.

Definition

LLM evaluation, often called evals, is how you check if a language model, or an app built on one, gives good answers. Most answers are free text, so there is no one right string to match. Teams build a set of test inputs and decide what a good answer looks like for each. The answers are then graded by code checks, by another model, or by people. Evals are run after each change to a prompt or model, so you can tell if things got better or worse.
A changed prompt runs through saved questions. The share of good answers falls, so the change is rolled back.

Common mistake

Trying a few questions by hand and calling it done. A handful of tries hides the cases where the model fails.

Key takeaways

  1. 1
    Free text answers. No single right string.
  2. 2
    Graded three ways. Code, model, people.
  3. 3
    Run after each change. Catch drops.

Learn it properly