⚙️Spark Executor
Also known as Executor (Spark), Spark Worker Process
A worker in Spark. It runs the tasks the driver sends it.
Think of it like
It is like a cook at one station in a big kitchen. The cook does the tasks the head chef sends.
Example
A cluster has 20 executors. Each one reads its own share of the files, so the job runs 20 ways at once.
Why it matters
Executors are where the time and the money go. Their number and size decide how fast a job runs.
Definition
Common mistake
Key takeaways
- 1Does the work. Runs the tasks it is sent.
- 2One machine each. Spread across the cluster.
- 3Keeps data. In memory or on disk between tasks.