Data Partitioning vs Database Sharding
People think partitioning and sharding are the same thing. Both split data into parts. One splits a table into parts inside one system, often by date, so queries can skip parts. The other spreads rows over many database servers, so each server takes part of the load.
Data Partitioning
A big table cut into parts by one column. Most often that column is a date.
Partitioning splits one big table into parts by the value of a column. The column is often a date, so each day or month gets its own part. In a data lake, each part is usually its own folder. Queries that filter on that column only read the parts they need. Old parts can also be dropped in one step.
Database Sharding
The rows of a big database, split across many servers. Each holds a part.
Sharding splits the rows of one big database across several servers. Each server, called a shard, holds only part of the data. A shard key, like the user id, decides which shard each row lives on. This lets a database grow past what one server can hold or handle. It adds a lot of work, so most teams shard only when nothing else is enough.
In one line
- Data Partitioning
- A big table cut into parts by one column. Most often that column is a date.
- Database Sharding
- The rows of a big database, split across many servers. Each holds a part.
These live in different corners of the glossary (Data and Development), which is exactly why they get confused.