Skip to content
BENIBIX Learn

Airflow Metadata Database

Data🗓️ OrchestrationApache Airflow

Also known as Airflow Metastore, Airflow Database

The database where Airflow keeps each run. It also holds task states and settings.

Think of it like

It is like the class book. Every name and every tick goes in it.

Example

After a year, a team's metadata database holds millions of old task rows. They clean out runs older than 90 days, and the web page is fast again.

Why it matters

It is the memory of the whole system. Lose it, and Airflow forgets what ran, what failed and how to reach your other systems.

Definition

The Airflow metadata database holds the state of everything in Airflow. It stores every DAG, every run and every task instance with its status. It also keeps connections, variables and XComs. The scheduler and the web page both read from it. PostgreSQL and MySQL are the usual choices in production. If it is slow or full, all of Airflow slows down.
Every arrow passes through this database. It is the one place that knows how each run went.

Common mistake

Putting big data in it, like large XComs. The pipeline's own data goes in a warehouse or a lake.

Key takeaways

  1. 1
    All state. DAGs, runs and task instances.
  2. 2
    Shared hub. The scheduler and web page read it.
  3. 3
    Keep it lean. Clean out old rows on a schedule.