Apache Airflow vs Dagster
People think Dagster is just another Airflow. Airflow plans the steps you run. Dagster plans the tables you want to exist, and works out the steps from them. So Dagster can tell you which table is stale. Airflow can tell you which task failed.
Apache Airflow
The top free tool for running data pipelines. It runs them on a clock.
Apache Airflow is a free, open source tool for orchestrating data pipelines. You write each pipeline as a DAG in Python code. A scheduler starts the runs on time and sends ready tasks to workers. A web page shows every run, every task and its logs. Thousands of ready-made operators talk to clouds, databases and tools. It was started at Airbnb in 2014.
Dagster
A free tool to run pipelines. It plans the work around the tables it must build.
Dagster is an open source orchestrator built around data assets. You declare the tables and files you want to exist, and how each is made from others. Dagster draws the graph from those links. It shows when each asset was last built and whether it is fresh. Tests and type checks run inside the pipeline. A paid cloud version runs it for you.
In one line
- Apache Airflow
- The top free tool for running data pipelines. It runs them on a clock.
- Dagster
- A free tool to run pipelines. It plans the work around the tables it must build.