Skip to content
BENIBIX Learn

MapReduce

Data⚙️ ProcessingDistributed Processing

Also known as Map Reduce, Hadoop MapReduce

Split the work. Many computers each do a part. Then join the results.

Think of it like

It is like a class counting a huge pile of beads. Each child counts a handful, then one child adds up all the counts.

Example

To count words in a billion web pages, each computer counts words in its own pages. Then all counts for "cat" go to one computer to be added up.

Why it matters

It showed how to use thousands of cheap computers as one. Spark and most big data tools grew from the idea.

Definition

MapReduce is a way to process huge data on many computers at once. In the map step, each computer works on its own slice and outputs key and value pairs. In the shuffle, pairs with the same key are sent to the same computer. In the reduce step, each computer combines the values for its keys. It was first described in 2004, and Hadoop made it open source.
Map works on each slice, the shuffle groups by key, and reduce adds up each group.

Common mistake

Writing new jobs in raw MapReduce. Spark and SQL engines do the same work faster and with far less code.

Key takeaways

  1. 1
    Map. Each computer works on its slice.
  2. 2
    Shuffle. Same key, same computer.
  3. 3
    Reduce. Combine each key's values.

Compare