# LLM-as-a-Judge

> One AI model grades another's answers. It is fast and cheap, but not perfect.

**Field:** Artificial Intelligence  
**Category:** Evaluation & Benchmarks  
**Also known as:** LLM Judge, AI Judge, Model-Graded Evaluation

## Think of it like

It is like a smart friend who marks your homework. They are fast, but not always right.

## Definition

LLM-as-a-judge means using one language model to grade the answers of another. The judge gets the question, the answer and a short guide, such as a score from 1 to 5 or a choice between two answers. It is much faster and cheaper than asking people, so it can grade thousands of answers. But judges have known biases. They often like longer answers, the first answer shown, or answers in their own style. So their grades should be checked against people's now and then.

## Example

Emma's evals use a judge model to grade 2,000 support answers overnight. Once a month, she grades 100 of them by hand and checks that she and the judge agree.

## Example code

```text
Rate the answer from 1 to 5 for how well it answers the question.
Question: {question}
Answer: {answer}
Reply with the number only.

4
```

A judge prompt, and the grade the judge model sends back.

## Why it matters

Free text answers are slow for people to grade. A judge makes it possible to test at scale.

## Common mistake

Trusting the judge without checking it. If you never compare its grades with people's, its biases become your score.

## Three principles

- **A model grades.** Another model's answers.
- **Fast and cheap.** Thousands at once.
- **Check against people.** Judges have biases.

## Related terms

- [LLM Evaluation](https://learn.benibix.com/term/ai/llm-evaluation)
- [Human Evaluation](https://learn.benibix.com/term/ai/human-evaluation)
- [Eval Set](https://learn.benibix.com/term/ai/eval-set)
- [Large Language Model (LLM)](https://learn.benibix.com/term/ai/large-language-model)
- [Eval Rubric](https://learn.benibix.com/term/ai/eval-rubric)
- [Pairwise Evaluation](https://learn.benibix.com/term/ai/pairwise-evaluation)
- [Evaluator-Optimizer Loop](https://learn.benibix.com/term/ai/evaluator-optimizer)

## Told apart from

- [LLM-as-a-Judge vs Human Evaluation](https://learn.benibix.com/compare/llm-as-a-judge-vs-human-evaluation): People often think an AI judge is just a cheaper way to get the grades people would give. It is faster and cheaper, and it can grade thousands of answers overnight. But it has biases of its own, such as liking long answers. Human evaluation is slow and costly, yet it is the standard. Most teams use the judge for scale and check it against people now and then.

![A question and an answer go to a judge model with a short guide. The judge sends back a grade of 4 out of 5.](https://learn.benibix.com/diagrams/ev-llm-as-a-judge.svg)

---

Source: https://learn.benibix.com/term/ai/llm-as-a-judge
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
