⚙️Data Shuffle
Also known as Shuffle, Spark Shuffle
Data is sent between computers. Rows with the same key end up in one place.
Think of it like
It is like a class sorting cards by colour. Each child walks their red cards to the red desk.
Example
A job groups a billion sales by store. One store has half the sales, so one computer works for an hour while the rest wait.
Why it matters
Shuffles decide how fast most big jobs run. Cutting one shuffle can halve a job's time.
Definition
Common mistake
Key takeaways
- 1Same key, same place. Needed for joins and group by.
- 2Slow by nature. Disk and network.
- 3Watch for skew. One big key slows all.