Skip to content
BENIBIX Learn

Model Parallelism

AI⚙️ MLOps & DeploymentLarge-Scale Training

Also known as Model Parallel Training, Tensor Parallelism, Pipeline Parallelism

A model too big for one GPU is cut into parts. Each GPU holds one part.

Think of it like

It is like a jigsaw too big for one table. You spread it over four tables.

Example

A model needs 300 GB of memory, but each of Lucy's GPUs has 80 GB. She splits its layers over 4 GPUs, so each holds about a quarter.

Why it matters

The biggest models do not fit on any single chip. Splitting them is how they get trained and served at all.

Definition

Model parallelism cuts a model that is too big for one GPU into parts. Each GPU holds only its own part. In pipeline style, each GPU holds some of the layers, and data passes from one GPU to the next. In tensor style, each layer itself is split across GPUs. The parts must talk all the time, so it needs very fast links.
The right panel is model parallelism. One model in four parts, and the data passes through each in turn.

Common mistake

Using it when the model fits on one GPU. Data parallelism is simpler and faster then.

Key takeaways

  1. 1
    Split the model. Each GPU holds one part.
  2. 2
    Pipeline style. Layers on different GPUs, in a row.
  3. 3
    Tensor style. Each layer split across GPUs.

Compare