Skip to content
BENIBIX Learn

Data Ingestion

Data📥 Ingestion & StreamingData Ingestion

Also known as Ingestion, Data Ingest

The first step of a pipeline. Data is brought in from where it is made.

Think of it like

It is like the loading dock of a shop. Every box comes in here first.

Example

Each hour, a job pulls new rows from the shop's database and new files from a partner's server. Both land in the data lake.

Why it matters

Every later step depends on it. If data is lost or doubled here, every report after it is wrong.

Definition

Data ingestion is the first step of a pipeline. It pulls data from where it is made into the place it will be kept. Sources can be app databases, files, APIs or streams of events. It can run in batches, such as once an hour, or nonstop as events happen. Nothing is cleaned much yet; the goal is to get the data in safely.

Common mistake

Copying the whole table each time. Just take the new rows. It is faster and costs less.

Key takeaways

  1. 1
    From the source. Databases, files, APIs and events.
  2. 2
    Batch or stream. On a timer or as it happens.
  3. 3
    Lose nothing. Every record must arrive once.