Skip to content
BENIBIX Learn

Apache Airflow

Data🗓️ OrchestrationApache Airflow

Also known as Airflow

The top free tool for running data pipelines. It runs them on a clock.

Think of it like

It is like a bus chart with a man at the stop. Buses leave in turn, and late ones are sent again.

Example

A shop runs 200 Airflow DAGs. They load data from apps, start dbt, and refresh dashboards before 7 am.

Why it matters

Many data teams use it as their main tool to run pipelines. Cloud firms sell it as a service, so the skill carries over.

Definition

Apache Airflow is a free, open source tool for orchestrating data pipelines. You write each pipeline as a DAG in Python code. A scheduler starts the runs on time and sends ready tasks to workers. A web page shows every run, every task and its logs. Thousands of ready-made operators talk to clouds, databases and tools. It was started at Airbnb in 2014.
Airflow's parts all meet at one database. The scheduler decides, the executor runs, and the web page shows.

Common mistake

Putting slow code at the top level of a DAG file. That code runs every time the file is read, not just when the pipeline runs.

Key takeaways

  1. 1
    Pipelines as code. Each DAG is a Python file.
  2. 2
    Huge library. Operators for most clouds and tools.
  3. 3
    See every run. A web page with status and logs.

Compare