Skip to content
BENIBIX Learn

Stream Processing

Data🧭 FoundationsData Pipelines

Also known as Streaming Data Processing, Real-Time Processing, Event Stream Processing

Each event is handled as it comes in. The answer is ready in seconds.

Think of it like

It is like a tap that is always on. You fill each cup the moment it is held out.

Example

A card company checks each payment as it happens. If a card is used in London and Tokyo within ten minutes, the second payment is blocked.

Why it matters

Some answers are worthless if they come late. Fraud, stock levels and live maps all need data within seconds.

Definition

Stream processing handles each piece of data as soon as it arrives. The data is a stream of small events, such as clicks or payments. A program reads the stream and acts on each event in seconds or less. It can count, filter or join events while they flow. It is fast, but it is harder to build and to fix than a batch.
A batch waits and runs on the pile. A stream handles each event as it comes.

Common mistake

Thinking stream means no delay at all. Events still come late or out of order, and the code has to handle that.

Key takeaways

  1. 1
    One event at a time. Each event is handled as it comes.
  2. 2
    Never done. The job runs all the time.
  3. 3
    Late events happen. The code must expect data out of order.

Compare

Learn it properly