⚙️Data Deduplication
Also known as Deduplication, Dedup, Dedupe
It takes out the extra copies of a row.
Think of it like
It is like a class list where one child got written down twice. You cross out the extra name.
Example
A retry loaded the same 400 orders twice. Deduplication on the order ID brings sales back to the true number.
Why it matters
Definition
Example code
sql
SELECT *
FROM orders
QUALIFY ROW_NUMBER() OVER (
PARTITION BY order_id ORDER BY loaded_at DESC
) = 1;Common mistake
Key takeaways
- 1Pick a key. What makes two rows the same.
- 2Keep one. Often the newest row.
- 3Expect copies. Retries always make some.