Data Parallelism vs Model Parallelism
People think data parallelism and model parallelism are one trick with two names. They are not. In the first, each GPU holds the whole model, and the data is split. In the second, the model is cut into parts, and each GPU holds one part. The first is simpler. The second is for a model too big for one GPU.
Data Parallelism
Each GPU holds the whole model. It trains on its own slice of the data.
Data parallelism is a way to train on many GPUs where each GPU holds a full copy of the model. The training data is split, and each GPU learns from its own share. After each step, the GPUs average what they learned, so all the copies stay the same. It is the simplest way to speed up training. It only works when the whole model fits on one GPU.
Model Parallelism
A model too big for one GPU is cut into parts. Each GPU holds one part.
Model parallelism cuts a model that is too big for one GPU into parts. Each GPU holds only its own part. In pipeline style, each GPU holds some of the layers, and data passes from one GPU to the next. In tensor style, each layer itself is split across GPUs. The parts must talk all the time, so it needs very fast links.
In one line
- Data Parallelism
- Each GPU holds the whole model. It trains on its own slice of the data.
- Model Parallelism
- A model too big for one GPU is cut into parts. Each GPU holds one part.