Skip to content
BENIBIX Learn

ROUGE Score

AI📏 Evaluation & BenchmarksText Generation Metrics

Also known as ROUGE, Recall-Oriented Understudy for Gisting Evaluation, ROUGE-L, ROUGE-N

A set of scores for summaries. They check how much of a person's summary was covered.

Think of it like

It is like checking a book report for the main points. You see how many of the teacher's points it has.

Example

Jack's app sums up meeting notes. He compares 200 of its summaries with ones his team wrote, and ROUGE-L goes from 0.38 to 0.45 after a prompt change.

Why it matters

Summaries are slow for people to grade. ROUGE gives a quick first check across hundreds of them.

Definition

ROUGE is a set of scores for summaries made by a computer. It compares the summary with summaries that people wrote. It counts how many words and short word runs from the human summary also show up in the computer's one. The focus is on recall, meaning how much of the human version was covered. ROUGE-N counts runs of N words, and ROUGE-L looks for the longest run of words in the same order. It was published in 2004.
ROUGE asks how much of the human version shows up in the model's output.

Common mistake

Chasing a higher ROUGE score alone. A summary can copy many words and still get the meaning wrong.

Key takeaways

  1. 1
    Recall first. How much of the human version is covered.
  2. 2
    Many kinds. ROUGE-N, ROUGE-L and more.
  3. 3
    Words, not meaning. Same idea in other words scores low.

Compare