🗓️Apache Airflow
Also known as Airflow
The top free tool for running data pipelines. It runs them on a clock.
Think of it like
It is like a bus chart with a man at the stop. Buses leave in turn, and late ones are sent again.
Example
A shop runs 200 Airflow DAGs. They load data from apps, start dbt, and refresh dashboards before 7 am.
Why it matters
Many data teams use it as their main tool to run pipelines. Cloud firms sell it as a service, so the skill carries over.
Definition
Common mistake
Key takeaways
- 1Pipelines as code. Each DAG is a Python file.
- 2Huge library. Operators for most clouds and tools.
- 3See every run. A web page with status and logs.