Data Sources and Data Silos
Level 1 · The field
Where a firm's data comes from, and why it gets stuck in separate tools. How data integration joins it into one source of truth.
Course contentsLesson 3 of 11
Level 3 · Kinds of data
Level 4 · Models and workloads
After this lesson you can
- List the usual data sources in a company
- Spot a data silo and say what it costs
- Explain what a single source of truth fixes
10 mincovers 5 glossary entries
Worth reading first: Data Engineering and Big Data
Where data comes from, and why it gets stuck
A data source is any system that makes or holds data you want. A small online shop might have five. There is its store, its payment tool, its support tool, its email tool and a few sheets. Each one holds datasets. A dataset is a named set of records, such as a table of orders.
A data silo is a source that keeps its data to itself. Nobody outside one team can see it, or it cannot be joined to anything else. Silos grow by accident. Each team buys the tool that suits it best, and nobody plans how the tools will share.
The cost shows up as arguments. Sales says 1,200 customers bought last month. Finance says 1,150. Both are right inside their own tool, because each one counts a little differently. A meeting about growth turns into a meeting about whose number is real.
Data integration is the work of joining sources into one view. It copies data from each source into a shared place. It matches records that mean the same thing, such as one customer with two emails. It also fixes the names, so the word sales means one thing in every report.
The aim is a single source of truth. That means one agreed place, and one agreed table, for each fact. Revenue comes from one table and nowhere else. When two reports disagree, you know which one to fix, because only one of them reads the agreed table.
A source of truth is a choice, not only a table. Someone has to agree that the finance count of customers wins. They write down why. Without that, a team builds a shiny new table, and it slowly turns into the next silo.
People often try to fix silos by buying one big tool that does it all. The old tools rarely go away, so now there is one more source. Joining the tools you have tends to work better than swapping them all out.
Worked example
Two numbers for one month
Emma's shop keeps orders in its store and refunds in its payment tool. Her sales for March are $48,000 in one and $45,500 in the other.
She copies both into one table and takes the refunds off once. Everyone now quotes $45,500, and the monthly sales meeting is 20 minutes shorter.
The words for this lesson5
Check yourself
What is a data silo?
Finished this lesson?
Marking it complete also marks its 5 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.