# Mixture of Experts (MoE)

> A model split into parts called experts. Each token uses only a few.

**Field:** Artificial Intelligence  
**Category:** LLMs  
**Also known as:** MoE, Sparse Mixture of Experts

## Think of it like

Think of it like a hospital. A desk sends each patient to the two doctors who fit best.

## Definition

Mixture of experts is a model design with many small networks, called experts, in some layers. A router sends each token to only a few of them. So the model can hold a huge number of weights, but each token uses only a small share. That makes it much cheaper to run than a model that uses every weight for every token. All the experts still have to sit in memory, though.

## Example

One well-known open model has 8 experts per layer and sends each token to 2. It holds about 47 billion weights but uses about 13 billion for each token.

## Why it matters

It lets very large models answer at the speed and cost of much smaller ones.

## Common mistake

Thinking it needs less memory. Only the compute drops; every expert must still be loaded.

## Three principles

- **Many experts.** Small networks in a layer.
- **A router picks.** A few per token.
- **Big but cheap to run.** Memory still holds all.

## Related terms

- [Transformer](https://learn.benibix.com/term/ai/transformer)
- [Parameter Count](https://learn.benibix.com/term/ai/parameter-count)
- [Layer (Neural Network)](https://learn.benibix.com/term/ai/neural-network-layer)

## Taught in

- [Transformer Architecture](https://learn.benibix.com/learn/ai-llms/transformer-architecture): LLMs, 5/15

![A token goes into a router. The router sends it to two of eight experts, which are lit up. The other six are idle for this token.](https://learn.benibix.com/diagrams/llm-moe.svg)

---

Source: https://learn.benibix.com/term/ai/mixture-of-experts
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
