Airflow vs dbt
People think dbt and Airflow do the same job. dbt turns raw tables into clean ones with SQL inside the warehouse. Airflow decides when jobs run and in what order. Airflow often starts dbt as one of its tasks. One does the work, and the other keeps the clock.
Apache Airflow
The top free tool for running data pipelines. It runs them on a clock.
Apache Airflow is a free, open source tool for orchestrating data pipelines. You write each pipeline as a DAG in Python code. A scheduler starts the runs on time and sends ready tasks to workers. A web page shows every run, every task and its logs. Thousands of ready-made operators talk to clouds, databases and tools. It was started at Airbnb in 2014.
dbt (Data Build Tool)
A tool that runs your SQL files. Each one builds a table, in the right order.
dbt is a tool for writing data transformations as SQL files. Each file is one SELECT that builds a table or view in the warehouse. Files point to each other with ref(), so dbt knows the order to run them in. It also runs tests on the results and writes docs from the code. dbt Core is free and open source; dbt Cloud is the paid service.
In one line
- Apache Airflow
- The top free tool for running data pipelines. It runs them on a clock.
- dbt (Data Build Tool)
- A tool that runs your SQL files. Each one builds a table, in the right order.