Skip to content
BENIBIX Learn

Batch Processing

Data🧭 FoundationsData Pipelines

Also known as Batch Job, Batch Pipeline

Handling data in big piles at set times. One job runs over all of it.

Think of it like

It is like doing the wash once a week. You wait for a full load, then wash it all at once.

Example

A bank adds up all of the day's card payments at 2am. The report is ready when staff get in at 9am.

Why it matters

Most reports do not need second-by-second data. A nightly batch is cheaper and easier to fix than a live stream.

Definition

Batch processing works on a big pile of data at once. Data builds up for a while, such as a day or an hour. Then a job runs and handles the whole pile in one go. When it ends, the results are ready to use. It is simple and cheap, but the answer is only as fresh as the last run.
A batch waits and runs on the pile. A stream handles each event as it comes.

Common mistake

Picking a stream when a batch would do. If people read the report once a day, a daily batch is enough.

Key takeaways

  1. 1
    Wait, then run. Data piles up before the job starts.
  2. 2
    All at once. One run handles the whole pile.
  3. 3
    As fresh as the last run. Results are as old as the last job.

Compare

Learn it properly