Skip to content
BENIBIX Learn

Spark Driver vs Spark Executor

People think the Spark driver does the heavy work, and it does not. The driver runs your code, plans the job and hands out tasks. The executors on other machines do the reading and the adding up. Only the final answer comes back to the driver. That is why pulling a huge table into it makes it crash.

Data⚙️ Processing

Spark Driver

The main Spark process. It plans the work and hands tasks to the executors.

The Spark driver is the main process of a Spark program. It runs your code, plans the work and splits it into small tasks. It then sends those tasks to executors on other machines and keeps track of them. When you ask for a result, such as with collect, the answer comes back to the driver. If the driver stops, the whole program stops.

Example
A job counts 2 billion clicks. The driver splits it into 400 tasks and sends them out. It gets back one small number.
Why it matters
Every Spark job has exactly one driver. When it runs out of memory, the whole job fails.
Full entry
Data⚙️ Processing

Spark Executor

A worker in Spark. It runs the tasks the driver sends it.

A Spark executor is a worker process that runs tasks for one Spark program. Each one runs on a machine in the cluster. It does the real work, such as reading files and adding up rows. It can also keep data in memory or on disk between tasks. More executors means more tasks can run at the same time.

Example
A cluster has 20 executors. Each one reads its own share of the files, so the job runs 20 ways at once.
Why it matters
Executors are where the time and the money go. Their number and size decide how fast a job runs.
Full entry

In one line

Spark Driver
The main Spark process. It plans the work and hands tasks to the executors.
Spark Executor
A worker in Spark. It runs the tasks the driver sends it.