⚙️Spark Structured Streaming
Also known as Structured Streaming, Spark Streaming
Spark's way to process streams, using the same code as batch jobs.
Think of it like
It is like a bus that leaves every few seconds. Each bus takes whoever is at the stop.
Example
A team already runs Spark batch jobs. They turn the nightly sales job into a stream that updates every 30 seconds.
Why it matters
One engine and one codebase serve both batch and stream. Teams on Spark need no new tool.
Definition
Example code
python
(clicks
.withWatermark("ts", "10 minutes")
.groupBy(F.window("ts", "5 minutes"), "page")
.count())Common mistake
Key takeaways
- 1Same code as batch. DataFrames over a stream.
- 2Micro-batches. Small batches, run again and again.
- 3Event time built in. Windows and watermarks.