Skip to content
BENIBIX Learn

Mixtral

Also known as Mixtral 8x7B, Mixtral 8x22B

Open models where each word goes to two of eight experts.

Think of it like

It is like a team of eight cooks. For each dish, only the two best cooks for it do the work.

Example

Lucy downloads Mixtral 8x7B from Mistral AI and runs it on her own server. Each word costs the work of about 13 billion weights, not 47.

Why it matters

It was one of the first open models of this kind. It did as well as much bigger models, and cost less to run.

Definition

Mixtral is a family of open-weight language models. Each one is built as a mixture of experts. Mixtral 8x7B came out in December 2023, with eight experts in each layer. A router sends each token to two of those eight. The model holds about 47 billion parameters, but each token uses only about 13 billion. So it runs at the speed of a much smaller model.
Mixtral works this way in every layer. Two of eight experts do the work for each token.

Common mistake

Reading 8x7B as 56 billion parameters. The experts share some parts, so the total is about 47 billion. Only about 13 billion work on each token.

Key takeaways

  1. 1
    Eight experts. In every layer of the model.
  2. 2
    Two per token. A router picks which two do the work.
  3. 3
    Open weights. Anyone can download and run it.

Learn it properly