Data Engineering and Big Data
Level 1 · The field
What data engineering is and why a company with data needs it. The six stages data passes through, and the point where data counts as big.
Course contentsLesson 1 of 11
Level 3 · Kinds of data
Level 4 · Models and workloads
After this lesson you can
- Say what a data engineer builds, and why
- Name the six stages of the data lifecycle
- Tell big data from data that fits on one machine
10 mincovers 4 glossary entries
What data engineering is, and what it is for
Data engineering is the work of moving data to where people use it. A shop makes data in three places. There is its website, its tills and its help desk inbox. A data engineer builds the pipes that bring all three into one place. Every report and every model sits on top of those pipes.
Data passes through six stages on that trip, and together they are called the data lifecycle. It is made, then brought in, then stored. Next it is cleaned and reshaped. Then it is served to the people and tools that use it. Last, it is archived or deleted. A data engineer owns the middle four.
The plan for how those stages fit together is the data architecture. It says which tool sits at each stage, where the data is kept, and who may read it. A small firm can draw it on one page. A bank may need a whole team just to keep that plan up to date.
Big data is data too large or too fast for one machine to handle. People describe it with three words that start with V. Volume is how much there is. Velocity is how fast it arrives. Variety is how many shapes it comes in, from tables to photos. Any one of the three can be enough.
The line moves every year, because one machine keeps getting stronger. A laptop today can sort a file of tens of millions of rows in seconds. Work that needed a room of servers fifteen years ago now often fits on one. So the honest test is simple: does the job still finish on one machine in time?
This course covers the ideas that every data job shares. It stops before the tools. Running a warehouse is a subject of its own. So is moving events with Kafka, and so is running jobs on a timer with Airflow. If you need tables and queries first, start with Databases and SQL.
People often mix up the jobs of data engineer and data scientist. A data scientist asks questions of the data and builds models. A data engineer makes sure the data is there before anyone asks. It has to be on time and right. When the pipes break, every chart down the line shows the wrong number.
Worked example
A bakery chain counts its sales
Lucy runs 4 bakeries. Each till saves a file at closing, and every Monday she pastes all 4 into one sheet. It takes her 3 hours, and one week she pastes Tuesday twice.
A small pipeline now copies each file into one table at midnight. Her Monday report is ready by 7am. No day is ever counted twice.
The words for this lesson4
Check yourself
When does data count as big data?
Finished this lesson?
Marking it complete also marks its 4 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.