Skip to content
BENIBIX Learn

Task Dependency

Data🗓️ OrchestrationWorkflow Orchestration

Also known as Upstream Task, Downstream Task, Pipeline Dependency

A rule that one task must finish before the next one starts.

Think of it like

You put on socks before shoes. The shoes have to wait for the socks.

Example

A report task depends on two load tasks. The loads run side by side, and the report starts only when both are done.

Why it matters

Links between tasks stop a report from reading half-loaded data. They also show which tasks can run side by side.

Definition

A task dependency says one task must finish before another can start. The task that goes first is upstream. The task that waits is downstream. Together the dependencies draw a graph with no loops, called a DAG. Tasks with no link between them can run at the same time. A task only runs when everything upstream has worked.

Example code

python
extract_orders >> join_tables
extract_payments >> join_tables
join_tables >> build_report
In Airflow, the arrow means the left task must finish first. The two extracts can run side by side.
The two extracts are upstream of the join. They share no arrow, so they run at the same time.

Common mistake

Chaining every task in one long line. Tasks that do not need each other wait for no reason, and the run takes twice as long.

Key takeaways

  1. 1
    Upstream first. The waiting task needs its inputs done.
  2. 2
    No loops. A task can never wait on itself.
  3. 3
    Parallel by default. Unlinked tasks run at the same time.