⚙️Processing
Pipelines and engines that reshape and crunch data · 39 terms
Data Transformation
11Changing raw data into the shape reports need: cleaning, removing copies, and the tool most teams write it in.
Raw data is changed into the shape a report needs.
It finds bad or missing values and fixes them.
It takes out the extra copies of a row.
Adding facts from a second list to your rows, like a city.
A tool that runs your SQL files. Each one builds a table, in the right order.
One SQL file in dbt. It builds one table or view.
A check that dbt runs on a table. It passes when no row breaks the rule.
A raw table listed in dbt, so models can read it by name.
A dbt table that keeps each old version of a row that changes.
A dbt model that adds just the new rows on each run.
Amazon's tool to find, clean and move data. It keeps a list of tables.
DataFrames
3Tables held inside a program, and the libraries that work on them in code.
Distributed Processing
12Splitting one big job over many computers, the systems that do it, and what slows it down.
Split the work. Many computers each do a part. Then join the results.
A free system for huge data. It runs on lots of cheap computers.
Hadoop's file store. It cuts big files up and keeps three copies.
A tool that puts SQL on top of Hadoop files.
A free tool that works on big data with lots of computers.
The way to use Apache Spark from Python.
The main Spark process. It plans the work and hands tasks to the executors.
A worker in Spark. It runs the tasks the driver sends it.
Data is sent between computers. Rows with the same key end up in one place.
When one part of a split job gets far more rows than the rest, so all wait.
A join that sends a small table to every worker. The big table stays put.
The AWS service that runs Spark and Hadoop clusters for you.
Streaming Engines
4Engines that work on events as they arrive and keep running totals.
A free tool that works on each event as soon as it comes in.
Spark's way to process streams, using the same code as batch jobs.
Write a pipeline once. Run it on Flink, Spark or Dataflow.
Google runs your Beam job for you. You never touch a machine.
Event Time and Windows
9How a stream knows when things happened, cuts time into windows, and copes with late events and crashes.
The time it really happened, not the time it came in.
The time the system handles an event, by its own clock.
A line that says no older events are expected now.
Events that come in after their window has closed.
Cutting an endless stream into pieces of time, so you can count.
Windows that are all one size. Each starts when the last ends.
Windows of one size that overlap. Each step gives a new result.
It stays open while events come. It shuts when they stop for a while.
It saves a stream job's state and place. Then a crash loses nothing.