Skip to content
BENIBIX Learn

Data Skew

Data⚙️ ProcessingDistributed Processing

Also known as Partition Skew, Skewed Data, Key Skew

When one part of a split job gets far more rows than the rest, so all wait.

Think of it like

It is like five friends sharing chores, but one gets all the dishes. Everyone waits for that one friend.

Example

A job groups sales by country on 50 machines. 80% of the sales come from one country, so one machine does most of the work while 49 wait.

Why it matters

It is a common reason a big job runs for hours instead of minutes. Adding machines does not help.

Definition

Data skew is when one part of a split job gets far more data than the others. Big jobs split rows into parts by a key, such as country. If most rows share one key, one machine gets most of the work. The rest finish fast and then sit waiting. The whole job is only as quick as its slowest part.
One worker holds 80% of the rows. The others finish early and wait for it.

Common mistake

Adding more machines to fix it. The busy part stays the same size, so the job is no faster.

Key takeaways

  1. 1
    Uneven parts. One key holds most of the rows.
  2. 2
    Slowest part wins. The job ends when the last part ends.
  3. 3
    Split the busy key. Add a random suffix to spread it out.

Compare