Spark Driver vs Spark Executor
People think the Spark driver does the heavy work, and it does not. The driver runs your code, plans the job and hands out tasks. The executors on other machines do the reading and the adding up. Only the final answer comes back to the driver. That is why pulling a huge table into it makes it crash.
Spark Driver
The main Spark process. It plans the work and hands tasks to the executors.
The Spark driver is the main process of a Spark program. It runs your code, plans the work and splits it into small tasks. It then sends those tasks to executors on other machines and keeps track of them. When you ask for a result, such as with collect, the answer comes back to the driver. If the driver stops, the whole program stops.
Spark Executor
A worker in Spark. It runs the tasks the driver sends it.
A Spark executor is a worker process that runs tasks for one Spark program. Each one runs on a machine in the cluster. It does the real work, such as reading files and adding up rows. It can also keep data in memory or on disk between tasks. More executors means more tasks can run at the same time.
In one line
- Spark Driver
- The main Spark process. It plans the work and hands tasks to the executors.
- Spark Executor
- A worker in Spark. It runs the tasks the driver sends it.