📥Data Ingestion
Also known as Ingestion, Data Ingest
The first step of a pipeline. Data is brought in from where it is made.
Think of it like
It is like the loading dock of a shop. Every box comes in here first.
Example
Each hour, a job pulls new rows from the shop's database and new files from a partner's server. Both land in the data lake.
Why it matters
Every later step depends on it. If data is lost or doubled here, every report after it is wrong.
Definition
Common mistake
Key takeaways
- 1From the source. Databases, files, APIs and events.
- 2Batch or stream. On a timer or as it happens.
- 3Lose nothing. Every record must arrive once.