Skip to content
BENIBIX Learn

Lemmatization

AI🔤 NLP & LanguageText Processing

Also known as Lemmatisation, Lemma

Each word goes back to its base form, so 'was' turns into 'be'.

Think of it like

Think of it like you look each word up. You find the plain form it comes from.

Example

Jack's tool reads 'The mice were running'. Lemmatization gives 'the mouse be run', so each word can be looked up.

Why it matters

It joins word forms that rules alone cannot, like 'went' and 'go', and it keeps real words.

Definition

Lemmatization turns each word into its dictionary form, called its lemma. Unlike stemming, it uses a word list and the word's job in the sentence. So 'studies' becomes 'study', 'was' becomes 'be', and 'better' becomes 'good'. The result is always a real word. It is slower than stemming, but more exact.
Lemmatization, in the last column, looks each word up. It gives back a real base word every time.

Common mistake

Thinking it is the same as stemming. Stemming chops by rule; lemmatization looks the word up.

Key takeaways

  1. 1
    Dictionary form. Called the lemma.
  2. 2
    Uses a word list. And the word's job.
  3. 3
    Always a real word. Slower, but exact.

Compare