Skip to content
BENIBIX Learn

Incremental Load

Data📥 Ingestion & StreamingData Ingestion

Also known as Incremental Ingestion, Delta Load, High-Water Mark

Copy only what is new since last time. Do not copy it all again.

Think of it like

You do not reread the whole book each night. You start from your bookmark.

Example

The orders table has 50 million rows. Each hour, the job copies only the few thousand that changed.

Why it matters

Big tables cannot be copied in full every hour. An incremental load keeps the job small and fast.

Definition

An incremental load copies only the rows that are new or changed since the last run. It keeps a mark of how far it got, often the latest updated time or the highest ID. The next run asks only for rows past that mark. A full load copies the whole table each time instead. Incremental loads are far faster on big tables.

Example code

sql
SELECT *
FROM orders
WHERE updated_at > '2026-11-15 23:00:00';
Ask only for rows changed after the mark the last run saved.

Common mistake

Using an updated time that the app does not always set. Rows changed without it are never picked up.

Key takeaways

  1. 1
    Keep a mark. Save how far the last run got.
  2. 2
    Ask past the mark. Only new or changed rows.
  3. 3
    Deletes need care. A deleted row has no new time.

Compare