Skip to content
BENIBIX Learn

Data-Aware Scheduling

Data🗓️ OrchestrationWorkflow Orchestration

Also known as Event-Driven Scheduling, Asset-Triggered Scheduling, Dataset Scheduling

A pipeline starts when its input data changes. It does not wait for a set time.

Think of it like

You eat when the food is on the table. You do not eat just because the clock says six.

Example

The orders team's load finishes at a different time each night. The finance report starts the moment it ends, never before.

Why it matters

Fixed times force you to add a safety gap. Waiting on the data itself removes the gap and the wrong reports it causes.

Definition

Data-aware scheduling starts a pipeline when the data it needs has changed. One pipeline marks a table or file as updated when it finishes. Any pipeline that reads that table then starts by itself. No one has to guess how long the first one takes. It links pipelines that belong to different teams.

Example code

python
from airflow.sdk import Asset, dag

orders = Asset("s3://lake/orders")

@dag(schedule=[orders])
def build_sales_report():
    ...
In Airflow 3, this pipeline runs each time a task updates the orders data.
Data-aware scheduling is on the right. The update to the table is itself the start signal.

Common mistake

Chaining twenty pipelines this way with no picture of the links. One slow source then stalls a whole tree, and no one sees why.

Key takeaways

  1. 1
    Data as the trigger. An updated table starts the next run.
  2. 2
    Across teams. Links pipelines that different people own.
  3. 3
    No guessed gaps. Runs start as soon as inputs land.