Skip to content
BENIBIX Learn

Bag of Words vs Word Embedding

People often think any way of turning words into numbers keeps their meaning. Bag of words only counts words, so 'car' and 'vehicle' are as far apart as 'car' and 'cake'. A word embedding gives each word a learned vector, so words with close meanings sit close. Counts are quick and clear; embeddings know which words are alike.

AI🔤 NLP & Language

Bag of Words

Text turned into word counts. The word order is thrown away.

Bag of words turns a text into a list of word counts. It makes one slot for every word in the vocabulary and counts how often each word shows up. The order of the words is thrown away. That makes it simple and fast, and it works well for jobs like sorting emails by topic. But 'dog bites man' and 'man bites dog' look exactly the same to it.

Example
Emma's spam filter counts words in each email. An email with 'free' 3 times and 'winner' 2 times gets a high spam score.
Why it matters
It was the first common way to turn text into numbers. It is still a quick test to beat.
Full entry
AI🔤 NLP & Language

Word Embedding

Each word as a list of numbers, so words with close meanings sit close.

A word embedding gives each word a vector, a list of a few hundred numbers. The numbers are learned from a corpus, so words used in the same way end up with close vectors. 'Cat' sits near 'dog', and far from 'tax'. Directions in the space can carry meaning too: king minus man plus woman lands near queen. Word2Vec, from 2013, made the idea popular.

Example
Lucy's search tool uses word embeddings. A search for 'car' also finds pages that only say 'vehicle', because the two vectors are close.
Why it matters
Embeddings let a computer tell how alike two words are. The language models of today are built on this idea.
Full entry

In one line

Bag of Words
Text turned into word counts. The word order is thrown away.
Word Embedding
Each word as a list of numbers, so words with close meanings sit close.