Skip to content
BENIBIX Learn

Airflow Catchup vs Backfill

People think catchup and a backfill are the same thing. Catchup is a switch on a DAG. When the DAG is turned on late, Airflow runs every missed date by itself, all at once. A backfill is a run you ask for, over dates you choose, when you are ready. Catchup happens to you. A backfill is something you plan.

Data🗓️ Orchestration

Airflow Catchup

Whether Airflow runs each missed day when a DAG is switched on. Or just the last one.

Catchup decides what Airflow does with runs it missed. A DAG may be switched on days after its start date. With catchup on, the scheduler makes one run for each missed time, oldest first. With catchup off, it makes only the latest run and skips the rest. In current Airflow, catchup is off unless you turn it on. You set it on each DAG.

Example
A daily DAG with a start date in January is switched on in March. With catchup on, Airflow makes about 60 runs at once. With it off, it makes one.
Why it matters
It decides whether one click starts one run or hundreds. Hundreds of runs at once can flood a database.
Full entry
Data📥 Ingestion & Streaming

Data Backfill

Run a pipeline again for past days. It fills gaps or fixes old data.

A backfill runs a pipeline over past dates to fill in or fix old data. It is needed when a new pipeline starts, so it has history to work with. It is also needed when a bug is fixed, so old wrong rows can be rebuilt. The job runs once for each missing day or hour. It only works well if the pipeline is safe to run twice.

Example
A bug cut tax from orders for three weeks. After the fix, the team backfills those 21 days, and the reports are right again.
Why it matters
Data is never right the first time forever. Backfills are how history gets repaired.
Full entry

In one line

Airflow Catchup
Whether Airflow runs each missed day when a DAG is switched on. Or just the last one.
Data Backfill
Run a pipeline again for past days. It fills gaps or fixes old data.