Skip to content
BENIBIX Learn

Lakeflow Pipelines

Data🧰 PlatformsData Platforms

Also known as Delta Live Tables, Lakeflow Spark Declarative Pipelines, Lakeflow Declarative Pipelines, Spark Declarative Pipelines

A Databricks tool that builds tables from the SQL you write. It was Delta Live Tables.

Think of it like

It is like a recipe card. You say what the dish should be, and the cook works out the steps.

Example

A team writes three short queries: raw orders, clean orders and daily totals. The pipeline builds them in the right order and drops 120 rows with no id.

Why it matters

You write what each table holds, not every step to build it. The tool handles the order and the new rows.

Definition

Lakeflow Pipelines is the Databricks tool for building pipelines by saying what each table should hold. It used to be called Delta Live Tables. You write one query in SQL or Python for each table, and the tool works out the order to build them in. A streaming table takes in new rows as they arrive. A materialized view keeps the answer to a query up to date. Rules called expectations can flag or drop bad rows.

Example code

sql
CREATE OR REFRESH STREAMING TABLE orders_clean (
  CONSTRAINT has_id EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW
) AS SELECT * FROM STREAM orders_raw;
One table and one rule. Rows with no order id are dropped before they land.
Two queries make two tables. The rule on the first one drops the rows with no id.

Common mistake

Thinking a plain expectation stops bad rows. On its own it only counts them. Add DROP ROW, or FAIL UPDATE to stop the run.

Key takeaways

  1. 1
    Declarative. Say what, not how.
  2. 2
    Two kinds of table. Streaming tables and materialized views.
  3. 3
    Expectations. Rules that warn, drop or fail.

Compare