Skip to content
BENIBIX Learn

Term Frequency-Inverse Document Frequency (TF-IDF)

AI🔤 NLP & LanguageText Representation

Also known as TF-IDF, TFIDF

A word score. It is high when a word is common in one page but rare in the rest.

Think of it like

Think of it like a word you see a lot in one book but not in the rest. It tells you what that book is about.

Example

Jack has 1,000 news stories. 'Said' is in nearly all of them, so it scores low everywhere. 'Penguin' is in 3, so it scores high in those 3.

Why it matters

It finds the words that make a document stand out, which is the core of simple search.

Definition

TF-IDF scores how important a word is to one document in a set. The first part, term frequency, counts how often the word appears in that document. The second part, inverse document frequency, lowers the score of words found in many documents. So 'the' scores near zero. But 'volcano' in a page about volcanoes scores high. Search engines used it for years to rank results.
A word in every document scores low. A word packed into just one scores high there.

Common mistake

Thinking a high score means the word is used most. A word used a lot everywhere still scores low.

Key takeaways

  1. 1
    Term frequency. How often in this document.
  2. 2
    Inverse document frequency. Lower if in many documents.
  3. 3
    Finds standout words. What makes a page different.