# Apache Spark

> A free tool that works on big data with lots of computers.

**Field:** Data Engineering  
**Category:** Processing  
**Also known as:** Spark, Spark SQL, Spark Cluster

## Think of it like

It is like a chef who splits a huge order among many cooks. Each cook keeps their food on the bench.

## Definition

Apache Spark is a free, open source engine for processing big data on many computers. A main program plans the work. It sends tasks to workers on each computer. It keeps data in memory between steps, which makes it much faster than MapReduce. You can use it from Python, SQL, Scala, Java or R. It handles batch jobs, streams and machine learning.

## Example

Each night, a Spark job runs on 40 computers. It joins 3 billion clicks with orders and writes the result to the lake.

## Why it matters

It is the most used engine for big data jobs. Databricks and most cloud data tools are built on it.

## Common mistake

Pulling a whole big table back to the driver with collect(). The driver runs out of memory and the job dies.

## Three principles

- **Driver and executors.** One plans, many do the work.
- **In memory.** Faster than MapReduce.
- **Many languages.** Python, SQL, Scala, Java, R.

## Related terms

- [PySpark](https://learn.benibix.com/term/data/pyspark)
- [Spark Driver](https://learn.benibix.com/term/data/spark-driver)
- [Spark Executor](https://learn.benibix.com/term/data/spark-executor)
- [Spark Structured Streaming](https://learn.benibix.com/term/data/spark-structured-streaming)
- [MapReduce](https://learn.benibix.com/term/data/mapreduce)
- [DataFrame](https://learn.benibix.com/term/data/dataframe)
- [Data Shuffle](https://learn.benibix.com/term/data/data-shuffle)

---

Source: https://learn.benibix.com/term/data/apache-spark
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
