Skip to content
BENIBIX Learn

Idempotent Pipeline

Data🗓️ OrchestrationWorkflow Orchestration

Also known as Idempotent Task, Idempotent Data Pipeline, Rerunnable Pipeline

You can run it twice for the same day. You still get the same result.

Think of it like

Pressing the lift button five times still brings one lift.

Example

Emma's job writes each day into its own slot and overwrites it on each run. She can rerun last month, and no sale is counted twice.

Why it matters

Every pipeline gets run again. A retry, a backfill or a person does it. If a rerun doubles the data, each time is a new bug.

Definition

An idempotent pipeline gives the same result however many times it runs for the same input. Running it twice for Monday leaves Monday's data exactly as one run would. The usual trick is to replace a whole slice instead of adding to it. For example, the task deletes Monday's rows, then writes them again. A merge on a key works too. This is what makes retries and backfills safe.

Example code

sql
DELETE FROM daily_sales
WHERE sale_date = '2026-11-02';

INSERT INTO daily_sales
SELECT * FROM staged_sales
WHERE sale_date = '2026-11-02';
Clear the day, then write it again. A second run leaves the same rows, not twice as many.
Run it twice. Adding doubles the day, while replacing leaves it right.

Common mistake

Using the time the job ran to decide which rows to take. A rerun next week then picks up a different set.

Key takeaways

  1. 1
    Replace, not add. Overwrite the slice the run owns.
  2. 2
    Fixed inputs. Pick rows by the run's date, not today.
  3. 3
    Safe reruns. Retries and backfills cannot double data.