⚙️Spark Driver
Also known as Driver Program, Spark Driver Program, Driver Node
The main Spark process. It plans the work and hands tasks to the executors.
Think of it like
It is like a head chef who reads the order and hands out the jobs. The chef does not chop.
Example
A job counts 2 billion clicks. The driver splits it into 400 tasks and sends them out. It gets back one small number.
Why it matters
Every Spark job has exactly one driver. When it runs out of memory, the whole job fails.
Definition
Common mistake
Key takeaways
- 1One per program. It runs your main code.
- 2Plans and sends. Splits the work into tasks.
- 3Gets results. Answers come back to it.