Video summary

AWS Glue Full Course For Beginners (2026 Step-By-Step Guide)

Main summary

Key takeaways

Educational

Main ideas & lessons conveyed

Purpose of the course / AWS Glue positioning

  • AWS Glue is presented as an “end-to-end one-stop” ETL/ELT solution plus a metadata/catalog foundation for aspiring AWS data engineers.
  • The speaker frames Glue as essential for building production-grade pipelines, including capabilities such as:
    • data cataloging and schema handling
    • crawlers
    • incremental loading
    • orchestration
    • Spark execution
    • triggers
    • integration with other AWS services (e.g., Lambda, IAM)

Big picture: what ETL/ELT pipelines really involve

The video emphasizes that ETL/ELT is not just “load then query.” Real pipelines typically include:

  • a staging/raw layer (often in S3)
  • catalog/metadata registration
  • transformation into presentation/warehouse-ready layers
  • scheduling and automation (time-based or event-based triggering)

Core AWS Glue definition (serverless data integration)

  • Glue is described as a serverless data integration service for:
    • discovering data
    • preparing data
    • moving data
    • integrating data from multiple sources
  • Glue also provides:
    • a centralized data catalog
    • tools for authoring/running/monitoring ETL jobs

Glue ecosystem & components

Major components mentioned:

  • Glue Data Catalog (backbone for table metadata and schemas)
  • Glue Crawlers
  • Create Table As Select (CTAS), including:
    • External tables
    • Managed tables
  • Glue Visual ETL (low/no-code pipeline building)
  • AWS Glue scripts / Spark scripts
  • AWS Glue Spark notebooks
  • Workflows (job orchestration)
  • Triggers & scheduling
  • Data quality
  • Incremental loading via bookmarks

Supporting concepts/services referenced repeatedly:

  • S3 (storage / data lake)
  • Athena (querying catalog-registered data)
  • IAM (permissions across Glue, S3, crawlers, triggers)
  • Lambda and EventBridge (event-based orchestration)

Methodologies / instruction-like content (detailed bullets)

1) Course setup & prerequisites

  • Ensure basic SQL and Python familiarity (explicitly stated as helpful).
  • Create an AWS account (free tier mention; educational credits mention).
  • Use the AWS Console, then:
    • Create an S3 bucket to act as the data lake for Glue-related data.
    • Create logical folders inside the bucket (e.g., raw/) and follow hierarchy best practices.

2) Create and use Glue Catalog (Data Catalog) basics

  • Store raw files (e.g., CSV) in S3 (typically under a folder like raw/).
  • In Glue Data Catalog:
    • Create a database (e.g., database_tutorial).
    • Register datasets as tables (often via external tables / CTAS examples).
  • Use the Glue Catalog as the abstraction layer:
    • CSV files in S3 are not “SQL tables” by default.
    • The Catalog registers metadata and schema so SQL engines (like Athena) can query them as tables.

3) Create External Tables (manual table registration)

Goal: Register table metadata in the Glue Catalog while keeping real data in user-managed S3.

Steps described (via Athena UI):

  • Select Data Catalog → choose a database.
  • Use Create table (UI-assisted wizard).
  • Configure:
    • S3 location (folder-level)
    • data format = CSV
    • delimiter = comma (or as appropriate)
  • Define column schema:
    • since CSV doesn’t carry schema, provide column names and types
    • the speaker references a “string-by-default then cast later” style approach
  • (Partitioning is noted as important, but deferred early on.)
  • Run CREATE TABLE to register the table.

Result

  • Query the CSV as a SQL table in Athena using the registered table name.

External table meaning (key rule)

  • The Catalog stores metadata.
  • Actual data remains in S3 under the user’s control.
  • Dropping the table metadata does not delete the S3 data.

4) CTAS with External vs Managed tables (CTAS-External / CTAS-Managed)

A) CTAS External

  • Create a new table using:
    • CTAS external + a SELECT query to define transformations/subsets
  • Provide:
    • table name
    • external location (target folder in S3)
    • format (e.g., Parquet)
  • Example logic shown:
    • create a derived table using a filter (e.g., “price > 1000” / “item = …”)
  • Result:
    • catalog registers metadata
    • data is written to the external location you choose

B) CTAS Managed

  • Use CTAS managed with a SELECT query and omit external location
  • Provide configuration context (bucket/URI selection via defaults/settings/database location)
  • Key rule / nuance:
    • dropping managed table metadata/data can remove underlying data in Glue-managed S3
    • behavior is tied to the Glue Catalog vs S3 separation
    • managed data location is auto-chosen by Glue, not necessarily where you think you pointed in S3

Practical experimentation suggestion (homework)

  • Create 3 external and 3 managed tables.
  • Drop each and observe how the corresponding S3 data behaves.

5) Glue Crawlers (schema inference & schema evolution)

A) Why crawlers

  • When raw files (e.g., CSV with no schema) are involved, crawlers infer schema automatically.
  • Crawlers can crawl:
    • files within folders
    • partition-style hierarchical folders (important for data lake layouts)

B) Crawlers require IAM permissions

  • Crawlers don’t automatically “see” S3 or update the Glue Catalog.
  • Attach an IAM role to the crawler that grants access to:
    • S3
    • Glue Catalog actions (and related glue execute permissions)

C) Schema evolution workflow (day1/day2 changes)

  • Day 1:
    • crawl files → create initial table schema in the catalog
  • Day 2:
    • add a new file (same folder) with an added column (e.g., cancelled_flag)
  • Configuration options discussed for schema changes:
    • Add new columns only
    • Ignore changes / don’t update table definition
    • Update all new and existing partitions with metadata
  • After rerunning the crawler:
    • catalog schema updates (depending on configuration)
    • older rows show null/empty values for newly introduced columns

6) Visual ETL (Glue Studio canvas) methodology

Setup

  • Open a Visual ETL job canvas.
  • Configure job details:
    • IAM role
    • Glue/Spark versions
    • worker type & number
    • optional components (bookmark, version control, data quality, etc.)

Pipeline construction on the canvas

  • Add a Source node (e.g., S3 raw folder)
    • specify CSV format
    • enable infer schema so downstream nodes receive schema
  • Add transformations:
    • Drop duplicates (framed as a good first transformation)
    • Type casting / schema change (important before math operations)
    • Custom transformation (Python-style custom logic)
    • SQL Query transformation
      • warning about DynamicFrame vs DataFrame compatibility
      • mentions needing a “select from collection” step
  • Add outputs/targets:
    • write transformed data back to S3 (e.g., Parquet)
    • optionally write multiple datasets (e.g., “clean” dataset and “aggregated” dataset)

Partitioning + data pruning instruction

  • Store outputs as partitioned data lake layouts by creating folders based on partition keys (e.g., item=...).
  • Benefit: data pruning, so queries scan only relevant partitions (e.g., only the mouse partition when filtering item = mouse).

7) Data quality checks inside Visual ETL

  • Add an Evaluate Data Quality node before writing targets.
  • Define rules such as:
    • Column count: pass/fail against expected number of columns
    • Column values: validate numeric conditions (e.g., total value >= 1000 when expected)
  • Configure outcomes:
    • whether to output the original input data
    • or output pass/fail results for logging/monitoring
  • Failure behavior:
    • speaker chooses to proceed with the “original data” flow while quality rules run.

8) Incremental loading (exactly-once style) using Glue bookmarks

  • Problem:
    • default pipelines reprocess all input files each run → duplicate ingestion
  • Solution:
    • enable job bookmarks for exactly-once-style ingestion

Conceptual steps

  • Create a new pipeline for incremental load.
  • Use a source S3 folder with files like day1.csv, day2.csv, etc.
  • Configure:
    • schema inference / table setup in the catalog (external table registration noted)
    • bookmarks enabled in job details
    • optionally support schema evolution via crawler updates

Validation approach

  • Run once → target contains day1 data
  • Add day2 file → rerun
  • Confirm:
    • only the new file is processed
    • only incremental new records are appended

9) Scheduling & triggering pipelines

A) Native scheduling (time-based / cron-like)

  • Create a schedule for the pipeline:
    • frequency (e.g., daily)
    • start hour
    • cron/chron expression behavior description

B) Event-based triggering

  • Use Lambda or EventBridge to trigger Glue jobs/workflows when events occur.
  • Lambda methodology described:
    • write Python Lambda code using boto3
    • call the Glue API to start the job (with correction: workflow run vs job run)
    • ensure Lambda has IAM permissions to start Glue jobs and access required resources

10) Workflows (orchestration of multiple jobs/crawlers)

  • Purpose:
    • manage dependencies and execution order across multiple Glue components
  • Key distinction:
    • Job = single pipeline/task unit
    • Workflow = orchestration graph linking multiple jobs/crawlers with dependencies
  • Workflow methodology shown:
    • create workflow canvas (e.g., a “parent workflow”)
    • add a start trigger (schedule or event)
    • add nodes with dependency triggers:
      • run job 1 (e.g., data migration)
      • then run crawler
      • then run incremental job
    • supports conditional/routing behavior (not fully shown in the demo)
  • Also mentioned:
    • passing global/job parameters into child components via workflow actions

11) Pro-code approaches in Glue

A) Spark scripts (ETL in code editor / script editor)

  • Create a role with permissions for:
    • S3 read/write
    • Glue execution
    • CloudWatch logging
    • plus inline policies as required
  • In the Glue script editor:
    • use Glue boilerplate (GlueContext / Spark session)
    • read CSV from S3 using parameterized paths:
      • --input_path
      • --output_path
    • write output back to S3 (e.g., Parquet/overwrite)
  • Configure job parameters (advanced properties) so the script receives arguments.

B) Spark notebooks (interactive development)

  • Create an interactive Glue notebook.
  • Steps:
    • start notebook session
    • run an initial cell to create GlueContext/Spark session
    • load from Glue Catalog (DynamicFrame from catalog)
    • convert to Spark DataFrame when needed
    • write output back to S3 (Parquet / Delta-like examples discussed)
  • Debugging demonstrated:
    • indentation/syntax errors
    • missing save before running
    • session authorization issues (including IAM pass role permission nuance)

12) AWS Glue DataBrew (data preparation tool overview)

  • Introduced as a visual data preparation product (conceptually similar to Power Query / Data Wrangler).
  • Intended for analytics and ML users.
  • Key idea:
    • build recipes (transformations) with no/low code using:
      • datasets
      • projects
      • recipes
  • Output:
    • create and run preparation jobs that produce cleaned/normalized datasets in S3.

Speakers / sources featured

  • Primary speaker: “Lamba” (instructor/host; referenced as AnLamba / Lamba / An Lamba)
  • Referenced external source: AWS Official Documentation (definition quoted for AWS Glue)
  • Referenced tool/source by name: ChatGPT (used by the speaker to suggest a random name during an ETL/ELT explanation)

Original video