Skip to content
BENIBIX Learn

Orchestration

Scheduling and coordinating multi-step workflows · 33 terms

Sort
Compare

Workflow Orchestration

10

Running pipeline steps in order and on time. The steps are written as code, and a person hears when one fails.

Pipeline steps run in the right order and on time. Failed steps get retried.

Writing the steps of a pipeline in a code file. No clicks in a tool.

A note to a person when a pipeline breaks or runs late.

Tasks and Dependencies

3

How steps wait for each other and for things outside the pipeline.

A rule that one task must finish before the next one starts.

One task splits into many that run at once. Then one task waits for them all.

A task that waits for a file or a table. Then the rest go on.

Schedules and Triggers

2

What starts a run: the clock, or the data it needs.

The part that starts each pipeline at the right time. It often uses a cron rule.

A pipeline starts when its input data changes. It does not wait for a set time.

Retries and Reruns

2

Running a step again after a failure without doubling the data.

A failed task runs again on its own. It gets a set number of tries.

You can run it twice for the same day. You still get the same result.

Apache Airflow

17

The free tool most data teams start with to run pipelines, and its parts. Also the clouds that run it for you.

The top free tool for running data pipelines. It runs them on a clock.

AWS runs Apache Airflow for you. You bring the DAG files, in an S3 bucket.

Google Cloud runs Apache Airflow for you. It used to be called Cloud Composer.

DAGs and Tasks

8

The pipeline file, its steps and their runs, and the notes they pass along. Also the missed runs it can catch up on.

One Airflow pipeline. It is a Python file with tasks, arrows and a time to run.

One run of an Airflow DAG. It is tied to the date of its data.

Whether Airflow runs each missed day when a DAG is switched on. Or just the last one.

One step in an Airflow DAG. It runs once in each DAG run.

Airflow tasks written as plain Python code, with @task on top.

A box that holds related Airflow tasks. Big DAGs get easier to read.

A ready-made Airflow template. Fill it in, and it becomes a task.

A small note one Airflow task leaves for another. It might be a file name or a count.

Airflow Components

3

The services that decide, run and remember every task.

The Airflow part that starts runs on time. It queues tasks that are ready.

The Airflow part that runs queued tasks. It can use one machine or many.

The database where Airflow keeps each run. It also holds task states and settings.

Airflow Integrations

3

Where Airflow keeps logins for outside systems, and the add-ons that talk to them.

A saved set of login details for an outside system. Tasks find it by name.

Code that logs in to one system for you, like a database. Tasks use it.

An add-on for Airflow. It brings the code to work with one outside system.

Orchestration Tools

6

Other tools that plan and run pipelines. Some are code, some are drag and drop, and some run on Kubernetes.

A free tool to run pipelines. It plans the work around the tables it must build.

A table or file in Dagster, described in code. The code also says how to build it.

A Python tool to run pipelines. Plain functions become flows that are tracked and retried.

A cloud tool from Microsoft. You drag boxes to copy data and run pipelines.

An AWS service that runs a workflow of steps. Each step calls a service.

A free tool that runs work on Kubernetes. Each step is a container.