# Data Parallelism

> Each GPU holds the whole model. It trains on its own slice of the data.

**Field:** Artificial Intelligence  
**Category:** MLOps & Deployment  
**Also known as:** Data Parallel Training, Data-Parallel Training

## Think of it like

It is like four cooks with the same recipe. Each one makes a quarter of the dinners.

## Definition

Data parallelism is a way to train on many GPUs where each GPU holds a full copy of the model. The training data is split, and each GPU learns from its own share. After each step, the GPUs average what they learned, so all the copies stay the same. It is the simplest way to speed up training. It only works when the whole model fits on one GPU.

## Example

Jack has 4 GPUs and a model that fits on one. He gives each GPU a quarter of each batch, and one pass over his data takes not much more than a quarter of the time.

## Why it matters

It is the first thing to try when training is too slow. Most code needs only small changes for it.

## Common mistake

Using it for a model too big for one GPU. Each GPU needs the whole model, so it runs out of memory.

## Three principles

- **Copy the model.** Every GPU holds all of it.
- **Split the data.** Each GPU learns from a slice.
- **Average each step.** So all the copies stay the same.

## Related terms

- [Distributed Training](https://learn.benibix.com/term/ai/distributed-training)
- [Model Parallelism](https://learn.benibix.com/term/ai/model-parallelism)
- [Batch Size](https://learn.benibix.com/term/ai/batch-size)
- [Gradient Descent](https://learn.benibix.com/term/ai/gradient-descent)
- [Video Memory (VRAM)](https://learn.benibix.com/term/infrastructure/video-memory)

## Told apart from

- [Data Parallelism vs Model Parallelism](https://learn.benibix.com/compare/data-parallelism-vs-model-parallelism): People think data parallelism and model parallelism are one trick with two names. They are not. In the first, each GPU holds the whole model, and the data is split. In the second, the model is cut into parts, and each GPU holds one part. The first is simpler. The second is for a model too big for one GPU.

![Two panels on top. Left: four GPUs, each with a full copy of the model, each fed a quarter of the data. Right: one model cut into four parts, one part on each GPU, with data passing through them in turn. Below, a timeline with save points, a crash, and training going on from the last save.](https://learn.benibix.com/diagrams/ai7-distributed-training.svg)

---

Source: https://learn.benibix.com/term/ai/data-parallelism
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-17
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
