# Apache Airflow

> The top free tool for running data pipelines. It runs them on a clock.

**Field:** Data Engineering  
**Category:** Orchestration  
**Also known as:** Airflow

## Think of it like

It is like a bus chart with a man at the stop. Buses leave in turn, and late ones are sent again.

## Definition

Apache Airflow is a free, open source tool for orchestrating data pipelines. You write each pipeline as a DAG in Python code. A scheduler starts the runs on time and sends ready tasks to workers. A web page shows every run, every task and its logs. Thousands of ready-made operators talk to clouds, databases and tools. It was started at Airbnb in 2014.

## Example

A shop runs 200 Airflow DAGs. They load data from apps, start dbt, and refresh dashboards before 7 am.

## Why it matters

Many data teams use it as their main tool to run pipelines. Cloud firms sell it as a service, so the skill carries over.

## Common mistake

Putting slow code at the top level of a DAG file. That code runs every time the file is read, not just when the pipeline runs.

## Three principles

- **Pipelines as code.** Each DAG is a Python file.
- **Huge library.** Operators for most clouds and tools.
- **See every run.** A web page with status and logs.

## Related terms

- [Workflow Orchestration](https://learn.benibix.com/term/data/workflow-orchestration)
- [Airflow DAG](https://learn.benibix.com/term/data/airflow-dag)
- [Airflow Scheduler](https://learn.benibix.com/term/data/airflow-scheduler)
- [Dagster](https://learn.benibix.com/term/data/dagster)
- [Prefect](https://learn.benibix.com/term/data/prefect)
- [Amazon MWAA](https://learn.benibix.com/term/data/amazon-mwaa)
- [Google Managed Service for Apache Airflow](https://learn.benibix.com/term/data/google-managed-service-for-apache-airflow)

## Told apart from

- [Airflow vs dbt](https://learn.benibix.com/compare/airflow-vs-dbt): People think dbt and Airflow do the same job. dbt turns raw tables into clean ones with SQL inside the warehouse. Airflow decides when jobs run and in what order. Airflow often starts dbt as one of its tasks. One does the work, and the other keeps the clock.
- [Apache Airflow vs Dagster](https://learn.benibix.com/compare/apache-airflow-vs-dagster): People think Dagster is just another Airflow. Airflow plans the steps you run. Dagster plans the tables you want to exist, and works out the steps from them. So Dagster can tell you which table is stale. Airflow can tell you which task failed.

![DAG files in a folder are read by a DAG processor into a metadata database. The scheduler reads the database, decides which tasks are ready and hands them to an executor, which runs them on workers. The web UI reads the same database.](https://learn.benibix.com/diagrams/airflow-architecture.svg)

---

Source: https://learn.benibix.com/term/data/apache-airflow
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-17
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
