⚙️Distributed Training
Also known as Multi-GPU Training, Parallel Training, Multi-Node Training
Training one model on many GPUs at once. A big job then takes days, not years.
Think of it like
It is like ten friends who fold one big pile of clothes. Together they are done fast.
Example
Lucy's team trains a model on 64 GPUs across 8 machines. A job that would take eight weeks on one machine finishes in under two weeks.
Why it matters
Today's large models are far too big and slow to train on one chip. Sharing the work is the only way.
Definition
Common mistake
Key takeaways
- 1Share the work. Many GPUs train one model.
- 2Two main splits. Split the data, or split the model.
- 3Links matter. GPUs swap results all the time.