Data Skew vs Data Shuffle
People think data skew and a data shuffle are the same slowdown, because both show up when a big job on many machines is slow. A shuffle is the step where rows move between machines, so rows with the same key meet. Skew is when one key has far more rows than the rest, so one machine gets most of them. A shuffle is often needed. Skew is a problem you can fix.
Data Skew
When one part of a split job gets far more rows than the rest, so all wait.
Data skew is when one part of a split job gets far more data than the others. Big jobs split rows into parts by a key, such as country. If most rows share one key, one machine gets most of the work. The rest finish fast and then sit waiting. The whole job is only as quick as its slowest part.
Data Shuffle
Data is sent between computers. Rows with the same key end up in one place.
A shuffle moves data between computers so rows with the same key end up together. A join, a group by or a sort on a big table needs one. It is often the slowest part of a job, because data is written to disk and sent over the network. If one key has far more rows than the rest, one computer gets most of the work. That is called skew.
In one line
- Data Skew
- When one part of a split job gets far more rows than the rest, so all wait.
- Data Shuffle
- Data is sent between computers. Rows with the same key end up in one place.