🛡️Great Expectations (GX)
Also known as GX, GX Core, Great Expectations Library
A free Python tool that tests data. It comes with lots of checks.
Think of it like
It is like a list of promises your data makes. Then someone checks each promise.
Example
A team runs GX on each new file from a partner. A file with a quantity below zero is sent back before it gets to the warehouse.
Why it matters
It gives Python pipelines one shared way to write and report data tests. The results also show what the data should look like.
Definition
Example code
python
import great_expectations as gx
context = gx.get_context()
source = context.data_sources.add_pandas("pandas")
asset = source.add_dataframe_asset(name="orders")
batch_def = asset.add_batch_definition_whole_dataframe("all")
batch = batch_def.get_batch(batch_parameters={"dataframe": df})
check = gx.expectations.ExpectColumnValuesToBeBetween(
column="quantity", min_value=1, max_value=100)
print(batch.validate(check).success)Common mistake
Key takeaways
- 1Expectations. Each check is a named promise.
- 2Ready-made library. Dozens of common checks built in.
- 3Readable results. Reports people can open and read.