Bag of Words vs Word Embedding
People often think any way of turning words into numbers keeps their meaning. Bag of words only counts words, so 'car' and 'vehicle' are as far apart as 'car' and 'cake'. A word embedding gives each word a learned vector, so words with close meanings sit close. Counts are quick and clear; embeddings know which words are alike.
Bag of Words
Text turned into word counts. The word order is thrown away.
Bag of words turns a text into a list of word counts. It makes one slot for every word in the vocabulary and counts how often each word shows up. The order of the words is thrown away. That makes it simple and fast, and it works well for jobs like sorting emails by topic. But 'dog bites man' and 'man bites dog' look exactly the same to it.
Word Embedding
Each word as a list of numbers, so words with close meanings sit close.
A word embedding gives each word a vector, a list of a few hundred numbers. The numbers are learned from a corpus, so words used in the same way end up with close vectors. 'Cat' sits near 'dog', and far from 'tax'. Directions in the space can carry meaning too: king minus man plus woman lands near queen. Word2Vec, from 2013, made the idea popular.
In one line
- Bag of Words
- Text turned into word counts. The word order is thrown away.
- Word Embedding
- Each word as a list of numbers, so words with close meanings sit close.