Skip to content
BENIBIX Learn

Spark Structured Streaming

Data⚙️ ProcessingStreaming Engines

Also known as Structured Streaming, Spark Streaming

Spark's way to process streams, using the same code as batch jobs.

Think of it like

It is like a bus that leaves every few seconds. Each bus takes whoever is at the stop.

Example

A team already runs Spark batch jobs. They turn the nightly sales job into a stream that updates every 30 seconds.

Why it matters

One engine and one codebase serve both batch and stream. Teams on Spark need no new tool.

Definition

Spark Structured Streaming is the part of Apache Spark that processes streams. You write the same DataFrame code you would write for a table. Spark runs it again and again on the new data, in small micro-batches by default. It supports event time, watermarks and exactly-once output to many sinks. A batch job can often become a stream with a few changes.

Example code

python
(clicks
  .withWatermark("ts", "10 minutes")
  .groupBy(F.window("ts", "5 minutes"), "page")
  .count())
Count clicks per page in 5-minute windows, and wait up to 10 minutes for late events.

Common mistake

Expecting a reply in a few thousandths of a second. Small batches add some delay, often a second or more.

Key takeaways

  1. 1
    Same code as batch. DataFrames over a stream.
  2. 2
    Micro-batches. Small batches, run again and again.
  3. 3
    Event time built in. Windows and watermarks.

Compare