Skip to content
BENIBIX Learn

Micro-Batching

Data🧭 FoundationsData Pipelines

Also known as Micro-Batch, Micro Batch Processing

Tiny batches run every few seconds. It feels close to live, but stays simple.

Think of it like

It is like a bus that leaves every five minutes. It does not wait to fill up. It does not go for just one person.

Example

Spark Structured Streaming works in micro-batches by default. A job can pick up new rows every ten seconds and add them to a total.

Why it matters

Many teams want fresh data but not the work of true streaming. Micro-batches give them most of the speed for less effort.

Definition

Micro-batching sits between batch and stream. Data is cut into very small batches, a few seconds' worth each. Each small batch is then run as a normal batch job. The result feels close to live, but the code stays as simple as a batch. The cost is a short, fixed delay.

Common mistake

Calling it real time. Each answer is at least one small batch old.

Key takeaways

  1. 1
    Small and often. Batches of a few seconds each.
  2. 2
    Batch code. Each tiny batch runs like a normal job.
  3. 3
    Small fixed delay. Results lag by about one batch.

Learn it properly