Skip to content
BENIBIX Learn

Stemming vs Lemmatization

People often think stemming and lemmatization are the same step. Both bring word forms back to one base, but in different ways. Stemming cuts endings off by rule, which is fast but can leave a part that is not a word. Lemmatization looks the word up, so 'was' becomes 'be' and the result is always a real word.

AI🔤 NLP & Language

Stemming

Cutting word endings off by rule, so 'jumps' and 'jumped' match.

Stemming cuts the endings off words so that different forms count as the same word. It follows simple rules, like dropping 'ing', 'ed' or 's'. That way 'connect', 'connected' and 'connecting' all become 'connect'. It is fast, but it does not check a dictionary, so the result is sometimes not a real word. A common English stemmer turns 'studies' into 'studi'.

Example
Emma searches her notes for 'running'. With stemming, notes that say 'runs' and 'run' show up too.
Why it matters
It helps search and word counts treat forms of one word as one, without much work.
Full entry
AI🔤 NLP & Language

Lemmatization

Each word goes back to its base form, so 'was' turns into 'be'.

Lemmatization turns each word into its dictionary form, called its lemma. Unlike stemming, it uses a word list and the word's job in the sentence. So 'studies' becomes 'study', 'was' becomes 'be', and 'better' becomes 'good'. The result is always a real word. It is slower than stemming, but more exact.

Example
Jack's tool reads 'The mice were running'. Lemmatization gives 'the mouse be run', so each word can be looked up.
Why it matters
It joins word forms that rules alone cannot, like 'went' and 'go', and it keeps real words.
Full entry

In one line

Stemming
Cutting word endings off by rule, so 'jumps' and 'jumped' match.
Lemmatization
Each word goes back to its base form, so 'was' turns into 'be'.