Skip to content
BENIBIX Learn

Apache Hadoop

Data⚙️ ProcessingDistributed Processing

Also known as Hadoop, Hadoop Cluster, Hadoop YARN

A free system for huge data. It runs on lots of cheap computers.

Think of it like

It is like a big team of workers with a shared shed. Each keeps some of the boxes and works on them where they sit.

Example

In the 2010s, a phone firm kept years of call logs on Hadoop. The cluster had 500 computers.

Why it matters

It started the big data age. Many of its ideas live on in Spark, data lakes and cloud storage.

Definition

Apache Hadoop is a free, open source system for storing and processing huge data on many cheap computers. It has three main parts. HDFS stores the files across the computers. YARN shares out the computers' power. MapReduce runs the jobs. It sends the work to the computers that hold the data, rather than moving the data.

Common mistake

Starting a new project on a Hadoop cluster. Cloud object storage and Spark now do the same job with far less to run.

Key takeaways

  1. 1
    HDFS. Files spread over many computers.
  2. 2
    YARN. Shares out the computers' power.
  3. 3
    Work goes to the data. Not the other way round.

Compare