# Transformer Architecture

> The design inside almost every LLM. How it turns tokens into lists of numbers, keeps word order, and passes them up through its layers.

**Course:** [Large Language Models (LLMs)](https://learn.benibix.com/learn/ai-llms)  
**Time:** 13 min

**After this lesson you can:**

- Name the parts of a transformer layer
- Explain why word order needs adding
- Tell encoder, decoder and MoE models apart

## What is inside a transformer

Almost every LLM today is a transformer. The design came from a 2017 paper by researchers at Google Brain. It reads all the tokens in a text at once, not one after another, so it learns far faster on big machines. GPT-3 stacked 96 of its layers, one on top of the next.

First, each token's ID number becomes a vector. A vector is a long list of numbers that stands for the token's meaning. Words used in similar ways end up with similar vectors. From here on, the model only works with these lists of numbers.

A transformer reads every token at the same moment, so on its own it has no sense of order. 'Dog bites man' and 'man bites dog' would look the same. Positional encoding fixes that. It adds a mark of each token's place to its vector, so word order counts.

Then come the layers. Each one does two jobs. Attention lets every token gather what it needs from the other tokens. Then a small network works on each token by itself. The output of one layer is the input to the next. At the top, the last vector becomes a chance for every token.

The first transformer had two halves. The encoder reads the whole input and builds a picture of it. The decoder writes the output one token at a time, and it only sees the tokens before the one it is writing. Translation used both: read the French, then write the English.

Later models split the halves. BERT, from 2018, kept only the encoder, and it is good at sorting and searching text. The GPT models kept only the decoder, and they are good at writing. Most chat models today are decoder-only.

A mixture of experts, or MoE, changes the small network in each layer. It holds many smaller networks, called experts, and a router sends each token to just a few. Mixtral 8x7B has 8 experts per layer and uses 2 per token. It holds about 47 billion parameters but uses about 13 billion per token.

People read 'expert' as a topic, like a maths expert or a history expert. It is not. The router learns its own split, and it often follows word shapes rather than subjects. The win is speed: a big model that does the work of a small one per token, though it still needs memory for all of it.

## The three shapes of transformer

| Shape | What it reads | Good at | Example |
| --- | --- | --- | --- |
| Encoder-only | The whole input at once | Sorting and searching text | BERT |
| Decoder-only | Only the tokens before | Writing text | GPT models |
| Encoder-decoder | The input, then writes | Translation | The 2017 original |

![Three tokens, The, cat and sat, go up through a stack: first vectors plus positions, then an attention step, then a small network, repeated over many layers, ending in a row of next-token chances.](https://learn.benibix.com/diagrams/llm-transformer.svg)

### Example: A travel app reads a German menu

A travel app turns German menus into English. Its encoder reads a whole line of the menu first, all 12 tokens at once.

Its decoder then writes the English one token at a time. Each time, it looks back at the encoder's picture of the German line.

**Words in this lesson:** [Transformer](https://learn.benibix.com/term/ai/transformer), [Positional Encoding](https://learn.benibix.com/term/ai/positional-encoding), [Attention Mechanism](https://learn.benibix.com/term/ai/attention-mechanism), [Encoder (Transformer)](https://learn.benibix.com/term/ai/encoder), [Decoder (Transformer)](https://learn.benibix.com/term/ai/decoder), [Mixture of Experts (MoE)](https://learn.benibix.com/term/ai/mixture-of-experts), [Gating Network](https://learn.benibix.com/term/ai/gating-network), [Mixtral](https://learn.benibix.com/term/ai/mixtral)

> **The one thing to remember.** Chat models use only the decoder half. They read all the text so far, then write the next token.

**Question:** What does positional encoding tell a model?

**Answer:** Where each token sits in the text. The model reads all tokens at once, so it has no order of its own. This step marks where each token sits. Then 'dog bites man' and 'man bites dog' differ.

---

Source: https://learn.benibix.com/learn/ai-llms/transformer-architecture
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
