# Idempotent Pipeline

> You can run it twice for the same day. You still get the same result.

**Field:** Data Engineering  
**Category:** Orchestration  
**Also known as:** Idempotent Task, Idempotent Data Pipeline, Rerunnable Pipeline

## Think of it like

Pressing the lift button five times still brings one lift.

## Definition

An idempotent pipeline gives the same result however many times it runs for the same input. Running it twice for Monday leaves Monday's data exactly as one run would. The usual trick is to replace a whole slice instead of adding to it. For example, the task deletes Monday's rows, then writes them again. A merge on a key works too. This is what makes retries and backfills safe.

## Example

Emma's job writes each day into its own slot and overwrites it on each run. She can rerun last month, and no sale is counted twice.

## Example code

```sql
DELETE FROM daily_sales
WHERE sale_date = '2026-11-02';

INSERT INTO daily_sales
SELECT * FROM staged_sales
WHERE sale_date = '2026-11-02';
```

Clear the day, then write it again. A second run leaves the same rows, not twice as many.

## Why it matters

Every pipeline gets run again. A retry, a backfill or a person does it. If a rerun doubles the data, each time is a new bug.

## Common mistake

Using the time the job ran to decide which rows to take. A rerun next week then picks up a different set.

## Three principles

- **Replace, not add.** Overwrite the slice the run owns.
- **Fixed inputs.** Pick rows by the run's date, not today.
- **Safe reruns.** Retries and backfills cannot double data.

## Related terms

- [Idempotency](https://learn.benibix.com/term/development/idempotency)
- [Task Retry](https://learn.benibix.com/term/data/task-retry)
- [Data Backfill](https://learn.benibix.com/term/data/data-backfill)
- [Data Partitioning](https://learn.benibix.com/term/data/data-partitioning)
- [Incremental Load](https://learn.benibix.com/term/data/incremental-load)

![Two rows of the same pipeline run twice for 2 November. On the top row, an append adds 100 rows each time and ends with 200. On the bottom row, a replace clears the day first and ends with 100.](https://learn.benibix.com/diagrams/idempotent-rerun.svg)

---

Source: https://learn.benibix.com/term/data/idempotent-pipeline
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-17
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
