Skip to content
BENIBIX Learn

Airflow Scheduler vs Airflow Executor

People think the Airflow scheduler runs the tasks. It only decides which tasks are ready and puts them in a queue. The executor takes them from the queue and runs them, on one machine or many. A slow queue and a slow task have different fixes.

Data🗓️ Orchestration

Airflow Scheduler

The Airflow part that starts runs on time. It queues tasks that are ready.

The Airflow scheduler is the part that decides what runs next. It makes new DAG runs when their time comes. It checks which tasks have all their upstream tasks done. It puts those tasks in a queue for the executor. It runs nonstop, and you can run more than one for safety. If it stops, nothing new starts.

Example
At 6:00, the scheduler makes today's run of the sales DAG. A few seconds later, the first two tasks are queued.
Why it matters
Every task in Airflow starts because the scheduler said so. When tasks sit queued for a long time, the scheduler is the first place to look.
Full entry
Data🗓️ Orchestration

Airflow Executor

The Airflow part that runs queued tasks. It can use one machine or many.

The Airflow executor decides where and how queued tasks run. The local one runs them as processes on the same machine. The Celery one sends them to a pool of worker machines. The Kubernetes one starts a fresh pod for each task. You pick one in Airflow's settings. The local one is the default.

Example
A small team runs Airflow on one server with the local kind. A bigger team uses the Kubernetes kind, so each task gets its own pod.
Why it matters
This part sets how many tasks run at once and how much memory each gets. Outgrowing it is the most common reason Airflow gets slow.
Full entry

In one line

Airflow Scheduler
The Airflow part that starts runs on time. It queues tasks that are ready.
Airflow Executor
The Airflow part that runs queued tasks. It can use one machine or many.