Apache Flink vs Spark Structured Streaming
People think Apache Flink and Spark Structured Streaming do the same thing at different speeds, and that is close but not all of it. Flink handles each event as it arrives, in milliseconds. Spark gathers events into tiny batches and runs the same code it uses for batch jobs. Pick Flink for speed and big state, and Spark to reuse what your team knows.
Apache Flink
A free tool that works on each event as soon as it comes in.
Apache Flink is a free, open source engine for processing streams of events one at a time. It works on event time, so late events still land in the right window. It keeps state, such as running totals, and saves it often in checkpoints. After a crash it starts from the last checkpoint, which gives exactly-once results. It also runs batch jobs and has a SQL layer.
Spark Structured Streaming
Spark's way to process streams, using the same code as batch jobs.
Spark Structured Streaming is the part of Apache Spark that processes streams. You write the same DataFrame code you would write for a table. Spark runs it again and again on the new data, in small micro-batches by default. It supports event time, watermarks and exactly-once output to many sinks. A batch job can often become a stream with a few changes.
In one line
- Apache Flink
- A free tool that works on each event as soon as it comes in.
- Spark Structured Streaming
- Spark's way to process streams, using the same code as batch jobs.