Skip to content
BENIBIX Learn

Lakeflow Pipelines vs dbt

People think Lakeflow Pipelines is Databricks' own dbt, and it is not quite. Both let you write SQL that says what each table should hold. dbt runs that SQL in batches, in almost any warehouse. Lakeflow Pipelines runs only in Databricks, and it can keep streaming tables fed as new rows arrive. Its rules can drop bad rows on the way in.

Data🧰 Platforms

Lakeflow Pipelines

A Databricks tool that builds tables from the SQL you write. It was Delta Live Tables.

Lakeflow Pipelines is the Databricks tool for building pipelines by saying what each table should hold. It used to be called Delta Live Tables. You write one query in SQL or Python for each table, and the tool works out the order to build them in. A streaming table takes in new rows as they arrive. A materialized view keeps the answer to a query up to date. Rules called expectations can flag or drop bad rows.

Example
A team writes three short queries: raw orders, clean orders and daily totals. The pipeline builds them in the right order and drops 120 rows with no id.
Why it matters
You write what each table holds, not every step to build it. The tool handles the order and the new rows.
Full entry
Data⚙️ Processing

dbt (Data Build Tool)

A tool that runs your SQL files. Each one builds a table, in the right order.

dbt is a tool for writing data transformations as SQL files. Each file is one SELECT that builds a table or view in the warehouse. Files point to each other with ref(), so dbt knows the order to run them in. It also runs tests on the results and writes docs from the code. dbt Core is free and open source; dbt Cloud is the paid service.

Example
A team keeps 120 SQL models in Git. Each night, dbt builds them in order and runs 300 tests.
Why it matters
It brings code habits to SQL: version control, tests, reviews and docs. That made it the usual tool for the T in ELT.
Full entry

In one line

Lakeflow Pipelines
A Databricks tool that builds tables from the SQL you write. It was Delta Live Tables.
dbt (Data Build Tool)
A tool that runs your SQL files. Each one builds a table, in the right order.