Skip to content
BENIBIX Learn

Spark Executor

Data⚙️ ProcessingDistributed Processing

Also known as Executor (Spark), Spark Worker Process

A worker in Spark. It runs the tasks the driver sends it.

Think of it like

It is like a cook at one station in a big kitchen. The cook does the tasks the head chef sends.

Example

A cluster has 20 executors. Each one reads its own share of the files, so the job runs 20 ways at once.

Why it matters

Executors are where the time and the money go. Their number and size decide how fast a job runs.

Definition

A Spark executor is a worker process that runs tasks for one Spark program. Each one runs on a machine in the cluster. It does the real work, such as reading files and adding up rows. It can also keep data in memory or on disk between tasks. More executors means more tasks can run at the same time.
The three boxes along the bottom are executors. Each one works on its own slice.

Common mistake

Giving executors too little memory. They spill data to disk or crash, and the job slows to a crawl.

Key takeaways

  1. 1
    Does the work. Runs the tasks it is sent.
  2. 2
    One machine each. Spread across the cluster.
  3. 3
    Keeps data. In memory or on disk between tasks.

Compare