Skip to content
BENIBIX Learn

Mixture of Experts (MoE)

AI💬 LLMsTransformer Architecture

Also known as MoE, Sparse Mixture of Experts

A model split into parts called experts. Each token uses only a few.

Think of it like

Think of it like a hospital. A desk sends each patient to the two doctors who fit best.

Example

One well-known open model has 8 experts per layer and sends each token to 2. It holds about 47 billion weights but uses about 13 billion for each token.

Why it matters

It lets very large models answer at the speed and cost of much smaller ones.

Definition

Mixture of experts is a model design with many small networks, called experts, in some layers. A router sends each token to only a few of them. So the model can hold a huge number of weights, but each token uses only a small share. That makes it much cheaper to run than a model that uses every weight for every token. All the experts still have to sit in memory, though.
The router lights up two of eight experts for this token. The other six do no work this time.

Common mistake

Thinking it needs less memory. Only the compute drops; every expert must still be loaded.

Key takeaways

  1. 1
    Many experts. Small networks in a layer.
  2. 2
    A router picks. A few per token.
  3. 3
    Big but cheap to run. Memory still holds all.

Learn it properly