Video summary

End to End Big Data Engineering Project with Azure | Big Data Engineering (From Scratch) | Part-1

Main summary

Key takeaways

Technology

End-to-End Big Data Engineering Project (Azure, Part 1) — Subtitle Summary

Project goal & dataset

  • Build a full end-to-end Big Data Engineering project from scratch using real-world e-commerce data.
  • Use the Olist (Brazilian e-commerce) dataset: “1 lakh orders”, including:
    • orders, customers, products, payments, reviews, and more
  • The dataset is split across multiple files/tables connected by keys like order_id and customer_id, so the pipeline must support multi-source joins and modeling complexity.

Tech stack (created/configured by the course)

  • Azure Data Factory (ADF): Orchestrates ingestion pipelines and copies data from multiple sources into storage.
  • Azure Data Lake Storage Gen2 (ADLS Gen2): Stores the data lake layers (Bronze/Silver/Gold).
  • Azure Data Bricks (Spark): Used later for transformations/enrichment (Part 2 referenced).
  • Azure Synapse: Used later for additional processing/transformation and serving tables.
  • MongoDB (NoSQL): Used for enrichment (example enrichment is described as increasing “realism/complexity”).
  • SQL database (MySQL via an external free hosting site “Files.io”): Simulates an industry SQL source.
  • HTTP/GitHub: One source is loaded via HTTPS requests to raw CSV files hosted on GitHub.

Prerequisites / setup guidance emphasized

  • Bring motivation and patience—don’t only watch; practice and implement.
  • Use a laptop + notebook; log errors and work through fixes.
  • Skills needed: Python + SQL (the course covers missing concepts, but expects baseline familiarity).
  • Azure setup highlights:
    • Create an Azure free account (“Try for free”) with a credit card for a small charge.
    • Clarify Azure credits model (e.g., “$200 credits”) and warn not to unintentionally upgrade to pay-as-you-go.
    • Explain the Azure account structure: Management Group → Subscription → Resource Group → Resources.
  • Choose an Azure region such as Central India.

Architecture implemented: Medallion (Bronze/Silver/Gold)

  • The project uses a medallion-style architecture:
    • Bronze: raw/ingested data
    • Silver: cleaned/filtered/aggregated/standardized data
    • Gold: business-ready serving/analytics layer
  • The approach is framed for interview readiness, including a “water analogy”:
    • raw → treated → final serving.

Data engineering workflow explained

Data Engineer responsibilities are framed as:

  • Ingest raw data from multiple sources
  • Store raw data in a data lake
  • Transform/clean/enrich
  • Build a serving layer for downstream analytics/data science

Data sources prepared (before ADF)

  1. HTTP/GitHub source

    • Reads multiple CSV files from a GitHub repository using the ADF HTTP connector (anonymous access).

  2. SQL source (external MySQL hosting)

    • Create a MySQL database and tables in the external SQL host.
    • Load CSV into MySQL using Python in Google Colab with:
      • pandas
      • mysql-connector
    • Includes practical error handling, e.g.:
      • Error: “no space left on device”
      • Fix: adjust batch insert size (e.g., reduce/modify batch size like 500).

  3. MongoDB source (NoSQL)

    • Later in the workflow:
      • Load an enrichment dataset into MongoDB via code (in Colab)
      • Later used with Azure Data Bricks

Azure Data Factory (ADF) ingestion pipeline (core tutorial)

Resource setup

  • Create an Azure Data Factory v2 resource under the project Resource Group.

ADF concepts emphasized

  • Pipelines with activities (visual drag-and-drop).
  • Link services:
    • Centralize connection settings to avoid re-entering credentials repeatedly.
  • Copy activity:
    • Copies from a configured source to a configured sink (ADLS Gen2).
  • Loops / iteration:
    • Uses ForEach to scale ingestion to many files without duplicating activities.
  • Lookup activity:
    • Fetches a JSON-like list to avoid manual input errors.
  • Parameterization + dynamic content:
    • Uses dynamic variables such as:
      • CSV relative URL
      • file name

Bronze layer loading (ADF → ADLS Gen2)

  • Create an ADLS Gen2 storage account and enable:
    • Hierarchical Namespace (to create folders like Bronze/Silver/Gold)
  • Create containers and directories:
    • bronze (Silver/Gold described for future steps)
  • Implement pipeline steps:
    1. Copy from HTTP/GitHub into ADLS Gen2 → Bronze
    2. Copy from SQL tables into ADLS Gen2 → Bronze
    3. Use ForEach with a JSON array describing the list of files/tables

Scalability design highlighted

  • Avoid a “copy-paste 7 times” approach.
  • Instead, define an input list (file/table definitions) as a JSON structure so that:
    • updating the JSON enables ingestion for new entries automatically.

Pipeline deployment behavior

  • Validate + debug using the ADF UI.
  • A publish step is required after validation/testing.

Folder/layer storage semantics

  • Uses ADLS containers and directory structure like:
    • bronze/<dataset_name>/...
  • Mentions storage tier concepts (Hot/Cool/Archive), with a practical note:
    • Archive tier may delay access and affect processing.

Azure Data Bricks setup (beginning, for Part 2)

  • Create an Azure Databricks workspace (trial-like configuration).
  • Explain Databricks conceptually as a Spark environment on Azure.
  • Discuss Databricks compute clusters:
    • Choose single-node for demo/free constraints
    • Mention runtime choices like Spark LTS
    • Note cluster termination settings to manage cost

MongoDB ingestion + enrichment rationale (preview)

  • Connect from notebooks (e.g., via Colab) to MongoDB and upload a dataset (example: product categories).
  • Explain why MongoDB is included:
    • Real projects often require external enrichment data
    • Example enrichment described in the same style as currency/dataset transformation, which Databricks would later process.

Big security/permissions preview (Data Bricks ↔ ADLS Gen2)

  • To read ADLS from Databricks using ABFSS, you need:
    • Azure Entra ID app registration (service principal)
    • Client secret
    • Role assignment (e.g., Storage Blob Data Contributor)
  • Permissions may take time to propagate:
    • wait approximately 15–20 minutes
  • Then read CSV from ADLS in Spark using ABFS/ABFSS-style locations and options.

Practical learning style

  • The presenter repeatedly emphasizes:
    • Expect errors and solve them (e.g., SQL batch insert issues, wrong loop variables, JSON comma mistakes, permission propagation delays).
    • Run and validate every step.
    • Use parameterization and automation patterns as done in real industry pipelines.

Main speakers / sources

Main speaker (host/presenter)

  • The instructor guiding through Azure resources, Azure Data Factory, Data Lake, and Databricks pipeline implementation.

Primary content sources referenced/used in the project

  • Olist e-commerce dataset (Brazil)
  • GitHub repository links (provided by the instructor)
  • External SQL/MySQL hosting site: “Files.io”
  • MongoDB (external/free-tier service)
  • Azure documentation concepts (implied through the way Azure resources and roles are explained)

Original video