Skip to content
BENIBIX Learn

Apache Beam

Data⚙️ ProcessingStreaming Engines

Also known as Beam, Beam SDK

Write a pipeline once. Run it on Flink, Spark or Dataflow.

Think of it like

It is like writing a song once. Any band can then play it. The words stay the same.

Example

A team writes a Beam pipeline that cleans sensor data. It runs on Dataflow in the cloud and on Flink in their own data centre.

Why it matters

It keeps a pipeline from being tied to one engine. Its model shaped how streaming is taught.

Definition

Apache Beam is a free, open source way to write one pipeline that works for both batch and streaming data. You write it once in Java, Python or Go. A runner then runs it, such as Flink, Spark or Dataflow. Beam brought the ideas of event time, windows, watermarks and triggers into one model.

Common mistake

Thinking each runner can do all of it. Some runners lack parts of Beam.

Key takeaways

  1. 1
    One model. Batch and stream alike.
  2. 2
    Many runners. Flink, Spark, Dataflow.
  3. 3
    Windows and triggers. Built into the model.