# Model Parallelism

> A model too big for one GPU is cut into parts. Each GPU holds one part.

**Field:** Artificial Intelligence  
**Category:** MLOps & Deployment  
**Also known as:** Model Parallel Training, Tensor Parallelism, Pipeline Parallelism

## Think of it like

It is like a jigsaw too big for one table. You spread it over four tables.

## Definition

Model parallelism cuts a model that is too big for one GPU into parts. Each GPU holds only its own part. In pipeline style, each GPU holds some of the layers, and data passes from one GPU to the next. In tensor style, each layer itself is split across GPUs. The parts must talk all the time, so it needs very fast links.

## Example

A model needs 300 GB of memory, but each of Lucy's GPUs has 80 GB. She splits its layers over 4 GPUs, so each holds about a quarter.

## Why it matters

The biggest models do not fit on any single chip. Splitting them is how they get trained and served at all.

## Common mistake

Using it when the model fits on one GPU. Data parallelism is simpler and faster then.

## Three principles

- **Split the model.** Each GPU holds one part.
- **Pipeline style.** Layers on different GPUs, in a row.
- **Tensor style.** Each layer split across GPUs.

## Related terms

- [Distributed Training](https://learn.benibix.com/term/ai/distributed-training)
- [Data Parallelism](https://learn.benibix.com/term/ai/data-parallelism)
- [Video Memory (VRAM)](https://learn.benibix.com/term/infrastructure/video-memory)
- [Model Memory Requirements](https://learn.benibix.com/term/ai/model-memory-requirements)
- [Layer (Neural Network)](https://learn.benibix.com/term/ai/neural-network-layer)

## Told apart from

- [Data Parallelism vs Model Parallelism](https://learn.benibix.com/compare/data-parallelism-vs-model-parallelism): People think data parallelism and model parallelism are one trick with two names. They are not. In the first, each GPU holds the whole model, and the data is split. In the second, the model is cut into parts, and each GPU holds one part. The first is simpler. The second is for a model too big for one GPU.

![Two panels on top. Left: four GPUs, each with a full copy of the model, each fed a quarter of the data. Right: one model cut into four parts, one part on each GPU, with data passing through them in turn. Below, a timeline with save points, a crash, and training going on from the last save.](https://learn.benibix.com/diagrams/ai7-distributed-training.svg)

---

Source: https://learn.benibix.com/term/ai/model-parallelism
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-17
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
