📏LLM-as-a-Judge
Also known as LLM Judge, AI Judge, Model-Graded Evaluation
One AI model grades another's answers. It is fast and cheap, but not perfect.
Think of it like
It is like a smart friend who marks your homework. They are fast, but not always right.
Example
Emma's evals use a judge model to grade 2,000 support answers overnight. Once a month, she grades 100 of them by hand and checks that she and the judge agree.
Why it matters
Free text answers are slow for people to grade. A judge makes it possible to test at scale.
Definition
Example code
text
Rate the answer from 1 to 5 for how well it answers the question.
Question: {question}
Answer: {answer}
Reply with the number only.
4Common mistake
Key takeaways
- 1A model grades. Another model's answers.
- 2Fast and cheap. Thousands at once.
- 3Check against people. Judges have biases.