Skip to content
BENIBIX Learn

Dagster

Data🗓️ OrchestrationOrchestration Tools

A free tool to run pipelines. It plans the work around the tables it must build.

Think of it like

Most planners list chores. This one lists the cakes you want, and works out the steps.

Example

A team moves from Airflow to Dagster. Now their page shows each table, when it was last built, and which ones are stale.

Why it matters

People care about tables, not tasks. Planning around assets makes it easy to see what data is fresh and what broke it.

Definition

Dagster is an open source orchestrator built around data assets. You declare the tables and files you want to exist, and how each is made from others. Dagster draws the graph from those links. It shows when each asset was last built and whether it is fresh. Tests and type checks run inside the pipeline. A paid cloud version runs it for you.

Example code

python
import dagster as dg

@dg.asset
def raw_orders():
    return load_orders()

@dg.asset
def daily_sales(raw_orders):
    return raw_orders.groupby("day").sum()
Two assets. Naming raw_orders as an input is enough for Dagster to draw the arrow.
Dagster draws the right-hand graph. Each box is a table you can check, not a step you ran.

Common mistake

Moving old task-based DAGs over one to one. You miss the main gain, which is modelling the tables themselves.

Key takeaways

  1. 1
    Assets first. Declare the tables you want to exist.
  2. 2
    Graph from inputs. Links come from what each asset reads.
  3. 3
    Freshness shown. See when each table last updated.

Compare