Skip to content
BENIBIX Learn

Spark Driver

Data⚙️ ProcessingDistributed Processing

Also known as Driver Program, Spark Driver Program, Driver Node

The main Spark process. It plans the work and hands tasks to the executors.

Think of it like

It is like a head chef who reads the order and hands out the jobs. The chef does not chop.

Example

A job counts 2 billion clicks. The driver splits it into 400 tasks and sends them out. It gets back one small number.

Why it matters

Every Spark job has exactly one driver. When it runs out of memory, the whole job fails.

Definition

The Spark driver is the main process of a Spark program. It runs your code, plans the work and splits it into small tasks. It then sends those tasks to executors on other machines and keeps track of them. When you ask for a result, such as with collect, the answer comes back to the driver. If the driver stops, the whole program stops.
The driver is the box at the top. It plans the work and sends tasks down.

Common mistake

Calling collect on a huge table. Every row is pulled into the driver, and it runs out of memory.

Key takeaways

  1. 1
    One per program. It runs your main code.
  2. 2
    Plans and sends. Splits the work into tasks.
  3. 3
    Gets results. Answers come back to it.

Compare