Airflow Catchup vs Backfill
People think catchup and a backfill are the same thing. Catchup is a switch on a DAG. When the DAG is turned on late, Airflow runs every missed date by itself, all at once. A backfill is a run you ask for, over dates you choose, when you are ready. Catchup happens to you. A backfill is something you plan.
Airflow Catchup
Whether Airflow runs each missed day when a DAG is switched on. Or just the last one.
Catchup decides what Airflow does with runs it missed. A DAG may be switched on days after its start date. With catchup on, the scheduler makes one run for each missed time, oldest first. With catchup off, it makes only the latest run and skips the rest. In current Airflow, catchup is off unless you turn it on. You set it on each DAG.
Data Backfill
Run a pipeline again for past days. It fills gaps or fixes old data.
A backfill runs a pipeline over past dates to fill in or fix old data. It is needed when a new pipeline starts, so it has history to work with. It is also needed when a bug is fixed, so old wrong rows can be rebuilt. The job runs once for each missing day or hour. It only works well if the pipeline is safe to run twice.
In one line
- Airflow Catchup
- Whether Airflow runs each missed day when a DAG is switched on. Or just the last one.
- Data Backfill
- Run a pipeline again for past days. It fills gaps or fixes old data.