Learn Data Engineering Basics
What data engineering is, and the ideas every data job shares. Eleven lessons start with the field, the people in it and where data comes from. Then come pipelines: ETL and ELT, batch and stream, and two designs that join them. After that, the shapes data comes in and how a model plans it. Last, why an app and its reports need different databases. It assumes nothing. It stops at the ideas, before any one tool.
Not started
0 of 11 lessons done
Level 1 · The field
score0250/14 wordsWhat data engineering is. Who does what in a data team. Where all the data comes from.
- Data Engineering and Big DataWhat data engineering is and why a company with data needs it. The six stages data passes through, and the point where data counts as big.10 min4 terms
- Data Roles and DataOpsThe four jobs in a data team, from data engineer to data scientist, and who does what. Then DataOps, the habits that stop their work from breaking.11 min5 terms
- Data Sources and Data SilosWhere a firm's data comes from, and why it gets stuck in separate tools. How data integration joins it into one source of truth.10 min5 terms
Level 2 · Pipelines
score25550/9 wordsHow data moves: ETL, ELT and reverse ETL, batch and stream, and two designs that join the two.
- ETL, ELT and Reverse ETLWhat a data pipeline is, and the three patterns it can follow. ETL cleans data before the warehouse. ELT cleans it inside. Reverse ETL sends it back out.12 min4 terms
- Batch and Stream ProcessingWhy some data is handled in one big run each night and some the moment it arrives. Where micro-batching sits between them, and what each one costs.11 min3 terms
- Lambda and Kappa ArchitectureTwo designs for answers that are both live and complete. Lambda runs a batch path and a stream path side by side, and Kappa runs a single stream.12 min2 terms
Level 3 · Kinds of data
score55700/5 wordsThe three shapes data comes in. Then two kinds of data with a time on every record.
- The Three Types of DataStructured, semi-structured and unstructured data, and how to tell them apart. What each one looks like, where it is kept, and why the shape matters.10 min3 terms
- Time Series and Event DataTwo kinds of data with a time on every record. Time series data comes on a steady beat. Event data is written each time something happens.9 min2 terms
Level 4 · Models and workloads
score701000/12 wordsHow a data model plans the data. When its shape gets checked. Why apps and reports split apart.
- Data Models and ER DiagramsWhat a data model is and how an ER diagram draws one. Then the three levels of a model, from a sketch for the business to the real tables.12 min5 terms
- Schema on Write and on ReadTwo moments to check the shape of data. Schema on write checks it as data is saved, schema on read checks it at query time. Then what changes break.11 min3 terms
- OLTP, OLAP and HTAPWhy the database behind an app is not the one behind its reports. OLTP serves many small writes, OLAP serves big reads, and HTAP tries to do both.12 min4 terms
Words in this course
The 40 glossary entries this course teaches, in the order it teaches them. Open any of them on its own.
- Data Engineering
- Data Lifecycle
- Data Architecture
- Big Data
- Data Engineer
- Analytics Engineer
- Data Analyst
- Data Scientist
- DataOps
- Data Source
- Dataset
- Data Silo
- Data Integration
- Single Source of Truth (SSOT)
- Data Pipeline
- Extract, Transform, Load (ETL)
- Extract, Load, Transform (ELT)
- Reverse ETL
- Batch Processing
- Stream Processing
- Micro-Batching
- Lambda Architecture
- Kappa Architecture
- Structured Data
- Semi-Structured Data
- Unstructured Data
- Time Series Data
- Event Data
- Data Model
- Entity-Relationship Diagram (ERD)
- Conceptual Data Model
- Logical Data Model
- Physical Data Model
- Schema-on-Write
- Schema-on-Read
- Schema Evolution
- Online Transaction Processing (OLTP)
- Online Analytical Processing (OLAP)
- Hybrid Transactional/Analytical Processing (HTAP)
- Denormalization