Skip to content
BENIBIX Learn

Gating Network

AI💬 LLMsTransformer Architecture

Also known as MoE Router, Expert Router, Router Network

The part that picks which experts each token goes to.

Think of it like

It is like the person at a front desk. They send each visitor to the right two rooms.

Example

In Emma's model, the router sees the token 'loop' and sends it to the two experts that score highest. The other six skip it.

Why it matters

The router is why a mixture of experts is cheap to run. Only the experts it picks do any work.

Definition

A gating network, or router, is the small part of a mixture of experts that picks the experts. For each token, it gives every expert a score. It keeps the top few, often one or two, and mixes their outputs by those scores. The other experts do no work for that token. In training, an extra rule pushes it to share tokens out evenly, so no expert sits idle.
The router is the box on the left. It scores all eight experts and sends the token to the top two.

Common mistake

Thinking each expert learns a clear subject, like maths or French. Studies of real routers find the split follows kinds of tokens, not neat topics.

Key takeaways

  1. 1
    Score every expert. One number each, for every token.
  2. 2
    Keep the top few. Often one or two.
  3. 3
    Share the load. Training keeps all experts busy.

Learn it properly