Skip to content
BENIBIX Learn

Apache Hadoop vs MapReduce

People think Hadoop and MapReduce are the same thing. MapReduce is a way to split a job into map and reduce steps over many computers. Hadoop is a whole system. It has a file store called HDFS, a part that shares out the computers, and a MapReduce engine. Spark can run on Hadoop with no MapReduce at all.

Data⚙️ Processing

Apache Hadoop

A free system for huge data. It runs on lots of cheap computers.

Apache Hadoop is a free, open source system for storing and processing huge data on many cheap computers. It has three main parts. HDFS stores the files across the computers. YARN shares out the computers' power. MapReduce runs the jobs. It sends the work to the computers that hold the data, rather than moving the data.

Example
In the 2010s, a phone firm kept years of call logs on Hadoop. The cluster had 500 computers.
Why it matters
It started the big data age. Many of its ideas live on in Spark, data lakes and cloud storage.
Full entry
Data⚙️ Processing

MapReduce

Split the work. Many computers each do a part. Then join the results.

MapReduce is a way to process huge data on many computers at once. In the map step, each computer works on its own slice and outputs key and value pairs. In the shuffle, pairs with the same key are sent to the same computer. In the reduce step, each computer combines the values for its keys. It was first described in 2004, and Hadoop made it open source.

Example
To count words in a billion web pages, each computer counts words in its own pages. Then all counts for "cat" go to one computer to be added up.
Why it matters
It showed how to use thousands of cheap computers as one. Spark and most big data tools grew from the idea.
Full entry

In one line

Apache Hadoop
A free system for huge data. It runs on lots of cheap computers.
MapReduce
Split the work. Many computers each do a part. Then join the results.