Skip to content
BENIBIX Learn

LLM-as-a-Judge

Also known as LLM Judge, AI Judge, Model-Graded Evaluation

One AI model grades another's answers. It is fast and cheap, but not perfect.

Think of it like

It is like a smart friend who marks your homework. They are fast, but not always right.

Example

Emma's evals use a judge model to grade 2,000 support answers overnight. Once a month, she grades 100 of them by hand and checks that she and the judge agree.

Why it matters

Free text answers are slow for people to grade. A judge makes it possible to test at scale.

Definition

LLM-as-a-judge means using one language model to grade the answers of another. The judge gets the question, the answer and a short guide, such as a score from 1 to 5 or a choice between two answers. It is much faster and cheaper than asking people, so it can grade thousands of answers. But judges have known biases. They often like longer answers, the first answer shown, or answers in their own style. So their grades should be checked against people's now and then.

Example code

text
Rate the answer from 1 to 5 for how well it answers the question.
Question: {question}
Answer: {answer}
Reply with the number only.

4
A judge prompt, and the grade the judge model sends back.
The question, the answer and a short guide go to the judge. It sends back a grade.

Common mistake

Trusting the judge without checking it. If you never compare its grades with people's, its biases become your score.

Key takeaways

  1. 1
    A model grades. Another model's answers.
  2. 2
    Fast and cheap. Thousands at once.
  3. 3
    Check against people. Judges have biases.

Compare