Skip to content
BENIBIX Learn

Bilingual Evaluation Understudy (BLEU)

AI📏 Evaluation & BenchmarksText Generation Metrics

Also known as BLEU Score, BLEU, BLEU Metric

A score for machine translation. It counts word runs that match a person's translation.

Think of it like

It is like checking a copy of a poem against the real one. You count how many word groups match.

Example

Emma tests two English to German models on 1,000 sentences with human translations. One scores 31 BLEU and the other 27, so she reads the first one's output more closely.

Why it matters

It lets teams compare translation models in seconds. No one has to read every line.

Definition

BLEU is a score for machine translation. It compares the computer's sentence with one or more translations made by people. It counts how many short runs of words match, from single words up to runs of four. It also takes points off when the output is too short. The score goes from 0 to 1 and is often shown out of 100. It came out in 2002 and is still used, because it is fast and free to run.
BLEU asks how much of the model's output appears in the human version.

Common mistake

Treating BLEU as a measure of meaning. A good translation that picks other words than the reference can still score low.

Key takeaways

  1. 1
    Word runs. Matches from one to four words long.
  2. 2
    Output side. How much of the output matches.
  3. 3
    Short penalty. Too short loses points.

Compare