Lakeflow Pipelines vs dbt
People think Lakeflow Pipelines is Databricks' own dbt, and it is not quite. Both let you write SQL that says what each table should hold. dbt runs that SQL in batches, in almost any warehouse. Lakeflow Pipelines runs only in Databricks, and it can keep streaming tables fed as new rows arrive. Its rules can drop bad rows on the way in.
Lakeflow Pipelines
A Databricks tool that builds tables from the SQL you write. It was Delta Live Tables.
Lakeflow Pipelines is the Databricks tool for building pipelines by saying what each table should hold. It used to be called Delta Live Tables. You write one query in SQL or Python for each table, and the tool works out the order to build them in. A streaming table takes in new rows as they arrive. A materialized view keeps the answer to a query up to date. Rules called expectations can flag or drop bad rows.
dbt (Data Build Tool)
A tool that runs your SQL files. Each one builds a table, in the right order.
dbt is a tool for writing data transformations as SQL files. Each file is one SELECT that builds a table or view in the warehouse. Files point to each other with ref(), so dbt knows the order to run them in. It also runs tests on the results and writes docs from the code. dbt Core is free and open source; dbt Cloud is the paid service.
In one line
- Lakeflow Pipelines
- A Databricks tool that builds tables from the SQL you write. It was Delta Live Tables.
- dbt (Data Build Tool)
- A tool that runs your SQL files. Each one builds a table, in the right order.