🧭Foundations
The core ideas behind moving and processing data · 40 terms
Data Engineering
9The field itself: the job, the people who do it and the path data takes.
The job of moving data and making it clean. Others then use it.
The stages data goes through, from being made to being deleted.
Data too big or too fast for one computer. Many machines share the work.
The big plan for a company's data. It shows where data lives and how it moves.
A way to test and ship data changes fast, so they do not break things.
Data Roles
4The jobs along the path of data, from bringing it in to asking it questions.
The person who builds the pipes that move data. They fix them when they break.
A person who studies data to answer questions. They share the answers in charts.
A person who cleans raw data into tested tables. Others build on those tables.
A person who uses math and code to find patterns. They often build models that guess.
Data Sources
5Where data comes from, the sets it arrives in, and how scattered copies are joined into one agreed answer.
Any place data comes from. It might be an app, a file or a sensor.
A set of data about one thing, kept together. Often it is one table.
Data stuck in one team's tool. Other teams cannot see it or use it.
Bringing data from many places into one place. Then it can be used as one.
One agreed place for each fact. Everyone reads the same number from it.
Data Pipelines
9Chains of steps that move data, and the ways they are ordered and timed.
A chain of steps that moves data from where it starts to where it ends up.
Pipeline Patterns
3Which way the data flows, and where the cleaning happens on the way.
Pull data out. Clean it on the way. Then load it in.
Load the raw data first. Then clean it in the warehouse.
Data goes back out of the warehouse. It lands in the tools teams use.
Batch and Stream Processing
5Handling data in piles or one event at a time, and designs that mix them.
Handling data in big piles at set times. One job runs over all of it.
Each event is handled as it comes in. The answer is ready in seconds.
Tiny batches run every few seconds. It feels close to live, but stays simple.
A slow, exact batch path and a fast stream path. The two are joined at the end.
One stream does all the work. Old data is played again when rules change.
Types of Data
5Data sorted by how much shape it has, from neat tables to loose files, and by how it is tied to time.
Data in neat rows and columns. Each column holds one kind of value.
Data with labels but no fixed shape. JSON is the best known kind.
Data with no rows, columns or labels. Text, photos, sound and video.
Time-Stamped Data
2Data where every row carries a moment: steady readings on a clock, or records of things as they happen.
Readings taken again and again over time. Each value has a time next to it.
A record of things that happened, one row each. Each row says what, when and who.
Data Modeling
9Planning what data to keep, how it links, and when its shape is set.
A plan of what data you keep and how the pieces link. It comes before the tables.
A drawing of boxes and lines. It shows the things you track and how they link.
The same data is copied into one wide table. Reads get much faster.
Changing the shape of data over time. Code that reads it must keep working.
Model Levels
3The same plan drawn at rising detail, from plain words to real tables.
The simplest data plan. Just the main things and how they link.
The data plan with every field and key added. It is not tied to any one database.
The data plan for one real database. It has exact column types and indexes.
Schema Approaches
2Whether data is checked against a shape when it is stored or when it is read.
The shape is fixed first. Data that does not fit is turned away.
Store data as it comes. The shape is set later, when it is read.
Data Workloads
3The kinds of work a database does, from quick app writes to big reports.
Many small, fast reads and writes. It is the work behind an app.
Big queries that add up lots of rows. It is the work behind reports.
One database does both jobs at once. It runs the app and the big reports.