Dense Vector vs Sparse Vector
People think a sparse vector is a weaker dense one, and it is not. A dense vector scores hidden traits of meaning, so it finds texts that say the same thing in other words. A sparse vector has one slot per word, so it finds exact words, names and codes. Each catches what the other misses, which is why hybrid search uses both.
Dense Vector
A list of numbers with a value in each slot. It holds meaning, not exact words.
A dense vector is a list of numbers where almost every number has a value. Embedding models make dense vectors. The numbers are not words. Each one scores a hidden feature of the meaning. So two texts with the same meaning get close vectors, even with no words in common. Searching with them is called dense retrieval.
Sparse Vector
A long list of numbers, mostly zeros. Each slot stands for one word.
A sparse vector is a very long list of numbers where nearly all are zero. Each slot stands for one word in a big vocabulary. Only the words in the text get a value, often a weight for how much that word matters. Keyword search methods like BM25 work this way. Sparse vectors are great at exact words, names and codes. They miss matches that use different words.
In one line
- Dense Vector
- A list of numbers with a value in each slot. It holds meaning, not exact words.
- Sparse Vector
- A long list of numbers, mostly zeros. Each slot stands for one word.