🧰Lakeflow Pipelines
Also known as Delta Live Tables, Lakeflow Spark Declarative Pipelines, Lakeflow Declarative Pipelines, Spark Declarative Pipelines
A Databricks tool that builds tables from the SQL you write. It was Delta Live Tables.
Think of it like
It is like a recipe card. You say what the dish should be, and the cook works out the steps.
Example
A team writes three short queries: raw orders, clean orders and daily totals. The pipeline builds them in the right order and drops 120 rows with no id.
Why it matters
You write what each table holds, not every step to build it. The tool handles the order and the new rows.
Definition
Example code
sql
CREATE OR REFRESH STREAMING TABLE orders_clean (
CONSTRAINT has_id EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW
) AS SELECT * FROM STREAM orders_raw;Common mistake
Key takeaways
- 1Declarative. Say what, not how.
- 2Two kinds of table. Streaming tables and materialized views.
- 3Expectations. Rules that warn, drop or fail.