Batch Processing vs Stream Processing
People think stream processing is always better because it is faster. A batch job runs on a pile of data at set times. A stream job handles each event as it comes. Streams cost more to build and to fix. Pick a stream only when a late answer is worth nothing.
Batch Processing
Handling data in big piles at set times. One job runs over all of it.
Batch processing works on a big pile of data at once. Data builds up for a while, such as a day or an hour. Then a job runs and handles the whole pile in one go. When it ends, the results are ready to use. It is simple and cheap, but the answer is only as fresh as the last run.
Stream Processing
Each event is handled as it comes in. The answer is ready in seconds.
Stream processing handles each piece of data as soon as it arrives. The data is a stream of small events, such as clicks or payments. A program reads the stream and acts on each event in seconds or less. It can count, filter or join events while they flow. It is fast, but it is harder to build and to fix than a batch.
In one line
- Batch Processing
- Handling data in big piles at set times. One job runs over all of it.
- Stream Processing
- Each event is handled as it comes in. The answer is ready in seconds.