Skip to content
BENIBIX Learn

Data Shuffle

Data⚙️ ProcessingDistributed Processing

Also known as Shuffle, Spark Shuffle

Data is sent between computers. Rows with the same key end up in one place.

Think of it like

It is like a class sorting cards by colour. Each child walks their red cards to the red desk.

Example

A job groups a billion sales by store. One store has half the sales, so one computer works for an hour while the rest wait.

Why it matters

Shuffles decide how fast most big jobs run. Cutting one shuffle can halve a job's time.

Definition

A shuffle moves data between computers so rows with the same key end up together. A join, a group by or a sort on a big table needs one. It is often the slowest part of a job, because data is written to disk and sent over the network. If one key has far more rows than the rest, one computer gets most of the work. That is called skew.

Common mistake

Joining a small table to a huge one with a full shuffle. Sending the small table to every computer is much faster.

Key takeaways

  1. 1
    Same key, same place. Needed for joins and group by.
  2. 2
    Slow by nature. Disk and network.
  3. 3
    Watch for skew. One big key slows all.

Compare