Skip to content
BENIBIX Learn

Google Cloud Dataflow

Data⚙️ ProcessingStreaming Engines

Also known as Dataflow, Cloud Dataflow, Google Dataflow

Google runs your Beam job for you. You never touch a machine.

Think of it like

It is like a kitchen you rent by the hour. You bring the recipe, and they bring the cooks.

Example

A game studio writes its event pipeline in Beam. Dataflow runs it all day. It adds workers each evening, when most people play.

Why it matters

It is the main reason teams pick Beam. On Google Cloud, it turns a Beam pipeline into a running job with nothing to set up.

Definition

Google Cloud Dataflow is a Google service that runs Apache Beam pipelines for you. You send it the pipeline, and it starts the machines, spreads the work and shuts them down after. It runs batch jobs and stream jobs with the same code. It adds or takes away workers as the load changes. You pay for the machines while the job runs.

Common mistake

Leaving a stream job running after it is no longer needed. Its workers keep billing every hour, even with no events.

Key takeaways

  1. 1
    Beam in. It runs pipelines written in Beam.
  2. 2
    No machines. Google starts and stops the workers.
  3. 3
    Grows with load. Workers are added when work piles up.