Skip to content
BENIBIX Learn

AWS Glue

Data⚙️ ProcessingData Transformation

Also known as Amazon Glue, AWS Glue Data Catalog, Glue Data Catalog, Glue

Amazon's tool to find, clean and move data. It keeps a list of tables.

Think of it like

It is like a librarian. She walks the shelves, writes a card for each book, and moves books where they are needed.

Example

Jack points a Glue crawler at a folder of sales files in S3. It adds a sales table to the catalog. A minute later, Athena can query it.

Why it matters

One catalog lets many AWS tools agree on what tables exist. The jobs run without servers to look after.

Definition

AWS Glue is a serverless tool from Amazon Web Services for finding and moving data. Its crawlers scan files and databases and guess each table's columns. The results go into the Glue Data Catalog, a shared list of tables. Athena and other tools can query it. Glue also runs ETL jobs on a Spark engine that AWS manages for you. It can connect to more than 70 kinds of data source.
A crawler fills the catalog. Athena and Glue jobs both read the same list of tables.

Common mistake

Trusting the crawler's guesses without checking. It can pick the wrong type for a column, such as text for a date.

Key takeaways

  1. 1
    Crawlers find. They guess tables from the data.
  2. 2
    Catalog lists. One shared list of tables.
  3. 3
    Jobs move. ETL runs on managed Spark.

Compare