⚙️Model Parallelism
Also known as Model Parallel Training, Tensor Parallelism, Pipeline Parallelism
A model too big for one GPU is cut into parts. Each GPU holds one part.
Think of it like
It is like a jigsaw too big for one table. You spread it over four tables.
Example
A model needs 300 GB of memory, but each of Lucy's GPUs has 80 GB. She splits its layers over 4 GPUs, so each holds about a quarter.
Why it matters
The biggest models do not fit on any single chip. Splitting them is how they get trained and served at all.
Definition
Common mistake
Key takeaways
- 1Split the model. Each GPU holds one part.
- 2Pipeline style. Layers on different GPUs, in a row.
- 3Tensor style. Each layer split across GPUs.