🏛️Mixtral
Also known as Mixtral 8x7B, Mixtral 8x22B
Open models where each word goes to two of eight experts.
Think of it like
It is like a team of eight cooks. For each dish, only the two best cooks for it do the work.
Example
Lucy downloads Mixtral 8x7B from Mistral AI and runs it on her own server. Each word costs the work of about 13 billion weights, not 47.
Why it matters
It was one of the first open models of this kind. It did as well as much bigger models, and cost less to run.
Definition
Common mistake
Key takeaways
- 1Eight experts. In every layer of the model.
- 2Two per token. A router picks which two do the work.
- 3Open weights. Anyone can download and run it.