📥Ingestion & Streaming
Getting data in and moving events as they happen · 35 terms
Data Ingestion
12Bringing data in from the places it is made, the connectors that do it, the ways to load it and the tools that run it.
The first step of a pipeline. Data is brought in from where it is made.
Ready-made code that copies data from one app or database. You set it up, not write it.
Run a pipeline again for past days. It fills gaps or fixes old data.
Ingestion Methods
6The ways data is copied in: all of it again, only what changed, from an app's API or event by event.
Copy the whole table again each time. Simple, but slow when the table is big.
Copy only what is new since last time. Do not copy it all again.
It copies each change in a database right as it is made.
A free CDC tool. It turns each database change into an event.
Pulling data out of an app through its API, a page at a time.
Bringing each event in as it happens. It lands in seconds, not once a night.
Ingestion Tools
3Products that copy data from apps, databases and streams into a warehouse or a lake.
A paid tool that brings app data into one big store.
A free tool whose code is open. It copies app data into a warehouse.
An AWS service that takes a stream and drops it in a store. No servers to run.
Event Streaming
7Moving events between systems as they happen, the services and platforms that carry them, the rules for their shape, and what happens when a reader falls behind.
Events move between systems as they happen. They are kept so they can be read again.
AWS's managed service for event streams. Streams are split into shards.
A Google tool. It sends one event to many readers.
A stream service in Microsoft's cloud. Kafka apps can use it as they are.
A free stream platform, first built at Yahoo. It keeps its data store apart.
A shared list of event shapes. It stops a writer from breaking readers.
A slow reader asks a fast writer to slow down. Then no data is lost.
Apache Kafka
12The open source event log most streams run on, the company that sells it as a service, and the parts it is built from.
The most used free tool for event streams.
A firm that sells Apache Kafka as a service. Kafka's makers started it.
A Kafka tool. It moves data in and out with ready plug-ins.
Kafka Clients
4The programs that write events into Kafka and read them out, and how readers share the work and keep their place.
A program that writes events into Kafka. It does not know who reads them.
A program that reads events from Kafka in order. An offset marks its place.
Readers that split a topic's partitions between them. Each group reads on its own.
The place number of an event in a partition. Readers save it like a bookmark.
Kafka Storage
5Where Kafka keeps events: the brokers, the topics and partitions, how long events stay and how many copies exist.
One server in a Kafka cluster. It holds partitions and serves reads.
A named stream of one kind of event, such as orders.
One ordered slice of a topic. More slices let more readers work at once.
How long a Kafka topic keeps its events before it deletes them. The default is 7 days.
Kafka keeps more than one copy of each partition. This number says how many.
Delivery Guarantees
4What a stream promises about a message: lost, sent twice or counted once when a send fails, and whether it arrives in order.
Each message comes once or not at all. Lost ones stay lost.
Every message comes, but some may come twice.
Each message counts one time, even if it is sent twice.
Whether events come in the order they were sent. Most only promise it per lane.