Airflow Scheduler vs Airflow Executor
People think the Airflow scheduler runs the tasks. It only decides which tasks are ready and puts them in a queue. The executor takes them from the queue and runs them, on one machine or many. A slow queue and a slow task have different fixes.
Airflow Scheduler
The Airflow part that starts runs on time. It queues tasks that are ready.
The Airflow scheduler is the part that decides what runs next. It makes new DAG runs when their time comes. It checks which tasks have all their upstream tasks done. It puts those tasks in a queue for the executor. It runs nonstop, and you can run more than one for safety. If it stops, nothing new starts.
Airflow Executor
The Airflow part that runs queued tasks. It can use one machine or many.
The Airflow executor decides where and how queued tasks run. The local one runs them as processes on the same machine. The Celery one sends them to a pool of worker machines. The Kubernetes one starts a fresh pod for each task. You pick one in Airflow's settings. The local one is the default.
In one line
- Airflow Scheduler
- The Airflow part that starts runs on time. It queues tasks that are ready.
- Airflow Executor
- The Airflow part that runs queued tasks. It can use one machine or many.