Stemming vs Lemmatization
People often think stemming and lemmatization are the same step. Both bring word forms back to one base, but in different ways. Stemming cuts endings off by rule, which is fast but can leave a part that is not a word. Lemmatization looks the word up, so 'was' becomes 'be' and the result is always a real word.
Stemming
Cutting word endings off by rule, so 'jumps' and 'jumped' match.
Stemming cuts the endings off words so that different forms count as the same word. It follows simple rules, like dropping 'ing', 'ed' or 's'. That way 'connect', 'connected' and 'connecting' all become 'connect'. It is fast, but it does not check a dictionary, so the result is sometimes not a real word. A common English stemmer turns 'studies' into 'studi'.
Lemmatization
Each word goes back to its base form, so 'was' turns into 'be'.
Lemmatization turns each word into its dictionary form, called its lemma. Unlike stemming, it uses a word list and the word's job in the sentence. So 'studies' becomes 'study', 'was' becomes 'be', and 'better' becomes 'good'. The result is always a real word. It is slower than stemming, but more exact.
In one line
- Stemming
- Cutting word endings off by rule, so 'jumps' and 'jumped' match.
- Lemmatization
- Each word goes back to its base form, so 'was' turns into 'be'.