Skip to content
BENIBIX Learn

Data Parallelism vs Model Parallelism

People think data parallelism and model parallelism are one trick with two names. They are not. In the first, each GPU holds the whole model, and the data is split. In the second, the model is cut into parts, and each GPU holds one part. The first is simpler. The second is for a model too big for one GPU.

AI⚙️ MLOps & Deployment

Data Parallelism

Each GPU holds the whole model. It trains on its own slice of the data.

Data parallelism is a way to train on many GPUs where each GPU holds a full copy of the model. The training data is split, and each GPU learns from its own share. After each step, the GPUs average what they learned, so all the copies stay the same. It is the simplest way to speed up training. It only works when the whole model fits on one GPU.

Example
Jack has 4 GPUs and a model that fits on one. He gives each GPU a quarter of each batch, and one pass over his data takes not much more than a quarter of the time.
Why it matters
It is the first thing to try when training is too slow. Most code needs only small changes for it.
Full entry
AI⚙️ MLOps & Deployment

Model Parallelism

A model too big for one GPU is cut into parts. Each GPU holds one part.

Model parallelism cuts a model that is too big for one GPU into parts. Each GPU holds only its own part. In pipeline style, each GPU holds some of the layers, and data passes from one GPU to the next. In tensor style, each layer itself is split across GPUs. The parts must talk all the time, so it needs very fast links.

Example
A model needs 300 GB of memory, but each of Lucy's GPUs has 80 GB. She splits its layers over 4 GPUs, so each holds about a quarter.
Why it matters
The biggest models do not fit on any single chip. Splitting them is how they get trained and served at all.
Full entry

In one line

Data Parallelism
Each GPU holds the whole model. It trains on its own slice of the data.
Model Parallelism
A model too big for one GPU is cut into parts. Each GPU holds one part.