🗓️Idempotent Pipeline
Also known as Idempotent Task, Idempotent Data Pipeline, Rerunnable Pipeline
You can run it twice for the same day. You still get the same result.
Think of it like
Pressing the lift button five times still brings one lift.
Example
Emma's job writes each day into its own slot and overwrites it on each run. She can rerun last month, and no sale is counted twice.
Why it matters
Every pipeline gets run again. A retry, a backfill or a person does it. If a rerun doubles the data, each time is a new bug.
Definition
Example code
sql
DELETE FROM daily_sales
WHERE sale_date = '2026-11-02';
INSERT INTO daily_sales
SELECT * FROM staged_sales
WHERE sale_date = '2026-11-02';Common mistake
Key takeaways
- 1Replace, not add. Overwrite the slice the run owns.
- 2Fixed inputs. Pick rows by the run's date, not today.
- 3Safe reruns. Retries and backfills cannot double data.