Skip to content
BENIBIX Learn

Data Backfill

Data📥 Ingestion & StreamingData Ingestion

Also known as Backfill, Backfilling

Run a pipeline again for past days. It fills gaps or fixes old data.

Think of it like

You missed a few days in your diary. So you go back and fill in those pages.

Example

A bug cut tax from orders for three weeks. After the fix, the team backfills those 21 days, and the reports are right again.

Why it matters

Data is never right the first time forever. Backfills are how history gets repaired.

Definition

A backfill runs a pipeline over past dates to fill in or fix old data. It is needed when a new pipeline starts, so it has history to work with. It is also needed when a bug is fixed, so old wrong rows can be rebuilt. The job runs once for each missing day or hour. It only works well if the pipeline is safe to run twice.

Common mistake

Backfilling a pipeline that appends rows. Each day ends up with two copies of every order.

Key takeaways

  1. 1
    Past dates. Rerun the job for old days.
  2. 2
    Fill or fix. New history or repaired history.
  3. 3
    Safe to rerun. The job must not double data.

Compare