aws-glue

Create a crawler to discover data schema

$ aws glue create-crawler --name [my-crawler] --role [arn:aws:iam::account:role/glue-role] --database-name [my-database] --targets S3Targets=[{Path=s3://my-bucket/data/}]

$ aws glue start-crawler --name [my-crawler]

$ aws glue get-databases

$ aws glue get-table --database-name [my-database] --name [my-table]

$ aws glue create-job --name [my-job] --role [arn:aws:iam::account:role/glue-role] --command Name=glueetl,ScriptLocation=s3://my-bucket/scripts/job.py

$ aws glue start-job-run --job-name [my-job] --arguments '{"--input_path":"s3://bucket/input","--output_path":"s3://bucket/output"}'

$ aws glue get-job-run --job-name [my-job] --run-id [jr_abc123]

aws glue command [options]

aws glue is the AWS CLI interface for AWS Glue, a serverless data integration service for ETL (extract, transform, load) workloads. Glue discovers, prepares, and combines data for analytics, machine learning, and application development.
Key components include the Data Catalog (central metadata repository), Crawlers (automatic schema discovery), Jobs (ETL scripts in Python or Scala), and Triggers (job orchestration). Glue integrates with S3, Redshift, RDS, and other data stores.

create-crawler

Create a crawler for schema discovery

Run a crawler to populate the catalog

List databases in the Data Catalog

List tables in a database

Get schema details for a table

Create an ETL job

Execute a job

Check job run status

Create a job trigger

List all crawlers

Crawlers can take significant time on large datasets. Job cold start adds latency; use job bookmarks for incremental processing. DPU (Data Processing Unit) costs accumulate during job runs. The Data Catalog has limits on tables per database (200,000).

AWS Glue launched in August 2017 as a serverless ETL service. Glue Studio for visual ETL authoring came in 2020. Data Quality features were added in 2022, and Glue for Ray (distributed Python) launched in 2023 for data science workloads.

aws(1), aws-athena(1), aws-s3(1), aws-redshift(1)