Batch and Stream Processing
Level 2 · Pipelines
Why some data is handled in one big run each night and some the moment it arrives. Where micro-batching sits between them, and what each one costs.
Course contentsLesson 5 of 11
Level 3 · Kinds of data
Level 4 · Models and workloads
After this lesson you can
- Tell batch, micro-batch and stream processing apart
- Match each one to how fresh an answer must be
- Explain why late data makes stream jobs hard
11 mincovers 3 glossary entries
Worth reading first: ETL, ELT and Reverse ETL
How batch, micro-batch and stream processing differ
Batch processing handles data in large groups on a schedule, such as once a night at 2am. Stream processing handles each piece of data the moment it arrives, often within a second. Micro-batching sits between them. It collects a few seconds of data, then handles that small group. The real choice is how long the answer may wait.
Batch is the simplest and the cheapest. A job wakes up, reads everything that came in since the last run, and writes the results. If it fails, you run it again. Most company reports are built this way, and a daily sales report rarely needs to be fresher than this morning.
Stream processing never stops. A program waits, and each new event flows through it as it arrives. It suits jobs where minutes matter, such as blocking a stolen card before the next purchase. It costs more to run and much more to get right, because the program must cope with trouble without pausing.
Micro-batching gives near-live results with batch tools. Data is gathered for a short window, often one to ten seconds, and then handled as a tiny batch. Results lag by a few seconds, not by a day. Spark first handled live data this way, and many teams still find it the easiest step up from batch.
The hard part of stream processing is that data arrives late and out of order. A phone in a tunnel sends ten minutes of taps all at once. A batch job does not care, because it reads the whole day later. A stream job has to decide how long to wait for stragglers before it counts.
People often pick stream processing because live sounds better. Ask who reads the result, and how often. If a person checks a chart once each morning, a nightly batch gives the same answer for a small part of the cost. Use live data where a machine acts on it within seconds.
Batch, micro-batch and stream side by side
| Way | How often it runs | How fresh the result is | Good for |
|---|---|---|---|
| Batch | On a schedule, often nightly | Hours old | Daily reports and bills |
| Micro-batch | Every few seconds | Seconds old | Live charts and near-live alerts |
| Stream | All the time, one event at a time | Under a second | Fraud checks and live prices |
Which one to pick
- 1
Batch
Pick it when a person reads the result once a day or less. It is cheap, simple and easy to run again.
- 2
Pick it when a few seconds of delay is fine and the team already knows batch tools.
- 3
Stream
Pick it when a machine must act on each event at once, such as stopping a bad payment.
Worked example
A bank checks card payments
Emma's bank ran a batch job at midnight to flag odd card payments. A thief with a stolen card had up to a day, and spent $900 on average before the flag.
A stream job now checks each payment in under a second. The next odd purchase is turned down at the till, before the thief can make a second one.
The words for this lesson3
Check yourself
When does stream processing beat a nightly batch?
Finished this lesson?
Marking it complete also marks its 3 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.