💬Mixture of Experts (MoE)
Also known as MoE, Sparse Mixture of Experts
A model split into parts called experts. Each token uses only a few.
Think of it like
Think of it like a hospital. A desk sends each patient to the two doctors who fit best.
Example
One well-known open model has 8 experts per layer and sends each token to 2. It holds about 47 billion weights but uses about 13 billion for each token.
Why it matters
It lets very large models answer at the speed and cost of much smaller ones.
Definition
Common mistake
Key takeaways
- 1Many experts. Small networks in a layer.
- 2A router picks. A few per token.
- 3Big but cheap to run. Memory still holds all.