Video summary

Azure Data Factory Full Course (From Beginner to PRO) | ADF Real-Time Scenarios

Main summary

Key takeaways

Educational

Main ideas / lessons conveyed

  • Audience goal: Become an Azure Data Factory (ADF) developer/pro from scratch by learning through hands-on, real-time scenarios that match interview expectations.
  • Why ADF matters:
    • Azure data engineers are in demand, and ADF is foundational (“backbone”) for orchestrating and moving data in Azure.
    • Even if you later work with Synapse or Microsoft Fabric (where ADF-like functionality exists), understanding ADF pipelines remains important because the concepts and UI/pipeline model are effectively reused.

Course structure (what you learn)

You cover:

  • Core ADF concepts and building blocks:
    • Pipelines
    • Activities
    • Connectors
  • ETL/EL concepts, including extract → load → transform
  • Data transformation using Data Flows (Spark-backed but GUI-driven)
  • Scheduling/orchestration triggers:
    • Schedule trigger
    • Tumbling window trigger (mentioned conceptually)
    • Storage event trigger (storage-blobs/events)
  • Complex orchestration using:
    • Parent/child pipelines
    • Execute Pipeline activity

Methodologies / instruction-like walkthroughs (detailed)

1) Prerequisites to start learning (course setup)

  • Use a laptop/PC with stable internet (iPad also acceptable if it can access Azure).
  • Create an Azure account (use “Try Azure for free” rather than “Pay as you go”).
  • Have excitement/enthusiasm to learn (motivational guidance).

2) What Azure Data Factory is (conceptual model)

  • ADF is described as a cloud ETL/EL tool.
  • Core workflow:
    • Extract data from sources (via connectors)
    • Load to destinations
    • Transform using Data Flows (GUI; Spark runs behind the scenes)

3) Key Azure building blocks created in the Azure portal

A) Resource Group

  • Acts like a folder to hold Azure resources.
  • Steps:
    1. In Azure portal → search/create Resource Group
    2. Provide:
      • Name (example: RG ADF course)
      • Region
    3. Skip tags (stated as out of scope for the moment)
    4. Create and verify it exists

B) Storage Account and Data Lake setup

  • Create a Storage Account (used as a backing store for Data Lake).
  • Steps:
    1. Azure portal → Create → search Storage Account
    2. Pick:
      • Resource Group
      • Storage Account name (must be globally unique)
      • Performance (standard used in example)
    3. Understand data redundancy options (cost vs security):
      • LRS (cheapest; within one datacenter)
      • ZRS, GRS, GZRS (more resilient; described as “more secure”)
    4. For the course: choose LRS (cost-focused guidance)
    5. Enable hierarchical namespace to make Data Lake (ADLS Gen2)-style storage behave like a Data Lake
    6. Create and verify resources deployment

C) Create Azure Data Factory (ADF)

  • Steps:
    1. Azure portal → Create → search Data Factory
    2. Name it (example: ADF course)
    3. Create and then open the ADF Studio later

4) ADF fundamentals: what you must know before building pipelines

A) Linked Service (connection)

  • A Linked Service is the connection/bridge between ADF and external services (sources/destinations).
  • Created for:
    • Storage accounts / Data Lake
    • GitHub via HTTP connector (example later)
    • Other external systems (conceptually)

B) Dataset (the actual data pointer)

  • A Dataset describes the data location and format inside the connected system:
    • Example: CSV file(s) inside a Data Lake container folder

5) Scenario 1 — Build your first pipeline (Copy Data Lake → Copy Data Lake)

  • Goal: Copy a CSV file from a source container to a destination container.
  • Components:
    • Linked Service to Storage/Data Lake
    • Source Dataset (CSV + folder/file selection)
    • Destination Dataset (CSV to destination container/folder)
    • Copy activity inside a Pipeline
  • Steps (as described):
    1. In ADF Studio → Manage tab: create Linked Service for Data Lake
    2. Upload a CSV file into source container (and create folder if needed)
    3. In Author tab:
      • New pipeline → add Copy activity
      • Name the activity (example: copy CSV)
      • Configure:
        • Source: linked service + dataset pointing to folder/file
        • Sink/destination: linked service + dataset pointing to destination folder
      • Enable header handling for CSV (first-row header)
    4. Run using Debug
    5. Validate by checking destination container for the new copied file

6) Scenario 2 — Copy from GitHub/HTTP to Data Lake (API/HTTP pull)

  • Goal: Pull a CSV directly from a GitHub raw URL into the destination Data Lake.
  • Components:
    • HTTP Linked Service
    • Source Dataset: HTTP dataset pointing to relative URL for the file
    • Destination Dataset: Data Lake CSV dataset
    • Copy activity
  • Steps:
    1. Create a new pipeline (example name like pipeline get)
    2. Copy activity:
      • Source uses HTTP connection type (recommended vs REST API for file resources)
      • Linked Service:
        • Base URL extracted from GitHub raw page
        • Authentication set to Anonymous (in example)
      • Dataset:
        • Relative URL set to the file path under the base URL
    3. Destination:
      • Data Lake linked service + CSV dataset pointing to destination folder
      • Note: leaving filename dynamic can create hierarchical folders
    4. Debug/run and validate the file in the destination container

7) Scenario 3 — Real-world data governance: route only “fact*” files to reporting

  • Problem described: The destination folder contains multiple files (e.g., fact_sales_1.csv and file_2.csv). Reporting/BI consumers must receive only compliant “fact” datasets.

  • Solution architecture:

    • Get Metadata activity to list files in a folder (child items: names/types)
    • ForEach activity to iterate over the list
    • If Condition activity to check filename:
      • startsWith(filename, "fact")
    • Inside the If-true branch:
      • Copy activity with parameterized datasets so the copy uses the current filename dynamically
    • Destination:
      • A new container like reporting

  • Steps (high level):

    1. Create a pipeline (example: only selected files)
    2. Add Get Metadata:
      • Dataset points to folder (not a specific file)
      • Output: child items array
    3. Add ForEach:
      • Iterate over getMetadataOutput.child items
    4. Add If condition:
      • Condition: starts with fact
    5. Add Copy activity in If-true branch:
      • Use parameterized source dataset:
        • Dataset parameter such as p_file_name
        • Bind parameter value from loop item name (item().name)
      • Destination dataset pointing to reporting container
    6. Run and verify: only fact-related files appear in the reporting folder

8) Scenario 4 — Transform with Data Flows (Spark-backed GUI transformations)

  • Goal: Transform reporting outputs and write transformed results back to Data Lake using Data Flows.
  • Transformation operations demonstrated (conceptual list):
    • Select columns (drop unwanted columns)
    • Filter rows (example: customer_id != 12)
    • Conditional split (split by payment type: Visa/MasterCard/Amex)
    • Derived column (replace nulls with "N/A" using conditional logic / coalesce-style behavior)
    • Aggregate / Group By (example: max product ID by customer ID)
  • Write step:
    • Use Sink to write to a destination folder
    • Mention: configure sink/row settings (example shows “always true” logic like 1=1)

9) Scenario 5 — Trigger orchestration: Scheduled trigger

  • Goal: Automatically run pipelines at a fixed cadence.
  • Steps:
    1. Create a pipeline that includes:
      • Get metadata → ForEach → If → Copy → Data Flow (triggered)
    2. Create a Schedule trigger:
      • Start time set to a near-future time
      • Recurrence interval specified (e.g., every 15 minutes)
      • Publish changes
    3. Validate via:
      • Manage → Triggers and Monitor to see pipeline runs

10) Scenario 6 — Trigger orchestration: Storage event trigger (event-driven ingestion)

  • Problem addressed: Scheduling alone isn’t enough because new files can arrive unpredictably.
  • Solution: Storage events trigger starts the pipeline when specific blobs/files appear.
  • Example behavior:
    • Trigger on blob created
    • Restrict to a specific path/filename pattern (e.g., source/CSV files/fact_sales_1.csv)
    • Pipeline deletes the file after successful copy to prevent retrigger loops
  • Steps:
    1. Create a trigger type: Storage events trigger
    2. Choose:
      • Subscription + storage account
      • Blob path starts with: container + folder + filename
    3. Event type: Blob created
    4. In the pipeline:
      • Add Delete activity on success after copy completes
    5. Address setup requirement:
      • Register required Azure provider/features (e.g., event grid support) if trigger activation fails
    6. Test:
      • Upload correct file → pipeline triggers and file is deleted
      • Upload wrong file → pipeline should NOT trigger

11) Scenario 7 — End-to-end orchestration using parent/child pipelines (Execute Pipeline)

  • Goal: Combine smaller pipelines into a single production workflow using Execute Pipeline.
  • Architecture:
    • A Parent pipeline triggers child pipelines in sequence:
      • Execute a “manager pipeline” (copy from source and GitHub, etc.)
      • Execute an “only selected files pipeline” (governance routing)
      • Execute additional pipelines as needed (e.g., git/missing part)
    • Attach a trigger (e.g., storage event trigger) at the parent pipeline level
  • Steps:
    1. Detach existing triggers from child pipelines as needed
    2. Create a Parent pipeline
    3. Add Execute Pipeline activities for child pipelines in sequence
    4. Add a trigger to parent pipeline (storage event based on file arrival)
    5. Publish and test:
      • Upload fact_sales_1.csv into the expected source path
      • Confirm all child pipelines run and data lands in correct containers

12) Advanced ADF concept: Set Variable activity

  • Purpose: Store outputs (like metadata arrays) into variables for later dynamic use.
  • Example flow:
    • After Get Metadata, use Set Variable to store child items array
    • Use the variable in subsequent logic (dynamic content)

Speakers / sources featured

  • Speaker: The video creator/instructor (referred to only by narration; no explicit name provided in subtitles).

Original video