⚙️Data Parallelism
Also known as Data Parallel Training, Data-Parallel Training
Each GPU holds the whole model. It trains on its own slice of the data.
Think of it like
It is like four cooks with the same recipe. Each one makes a quarter of the dinners.
Example
Jack has 4 GPUs and a model that fits on one. He gives each GPU a quarter of each batch, and one pass over his data takes not much more than a quarter of the time.
Why it matters
It is the first thing to try when training is too slow. Most code needs only small changes for it.
Definition
Common mistake
Key takeaways
- 1Copy the model. Every GPU holds all of it.
- 2Split the data. Each GPU learns from a slice.
- 3Average each step. So all the copies stay the same.