Skip to content
BENIBIX Learn

Apache Spark

Data⚙️ ProcessingDistributed Processing

Also known as Spark, Spark SQL, Spark Cluster

A free tool that works on big data with lots of computers.

Think of it like

It is like a chef who splits a huge order among many cooks. Each cook keeps their food on the bench.

Example

Each night, a Spark job runs on 40 computers. It joins 3 billion clicks with orders and writes the result to the lake.

Why it matters

It is the most used engine for big data jobs. Databricks and most cloud data tools are built on it.

Definition

Apache Spark is a free, open source engine for processing big data on many computers. A main program plans the work. It sends tasks to workers on each computer. It keeps data in memory between steps, which makes it much faster than MapReduce. You can use it from Python, SQL, Scala, Java or R. It handles batch jobs, streams and machine learning.

Common mistake

Pulling a whole big table back to the driver with collect(). The driver runs out of memory and the job dies.

Key takeaways

  1. 1
    Driver and executors. One plans, many do the work.
  2. 2
    In memory. Faster than MapReduce.
  3. 3
    Many languages. Python, SQL, Scala, Java, R.