Video summary

How to Design a Production-Grade System in Python

Main summary

Key takeaways

Technology

Overview

The video explains how to build a production-grade, large-scale web scraping system in Python for an Amazon price comparison agent. The system is designed to run continuously, avoid blocks, scrape across multiple countries, store historical data, and support AI-based querying (instead of manual SQL).

Use Case: Amazon Price Competitor Tool

The system is designed to build an Amazon price competitor tool that:

  • Accepts an ASIN/product ID
  • Scrapes the product from multiple Amazon country domains/regions
  • Collects prices and product details (e.g., title, brand, category)
  • Enables real-time price discrepancy monitoring across regions

Why basic scrapers fail at scale

Basic scrapers break down due to:

  • IP blocking / CAPTCHA
  • Country/region page differences
  • Session instability
  • Inconsistent data, reducing reliability and trust

As a result, scraping becomes a systems engineering problem involving reliability, retries, orchestration, and failure handling.

High-Level Architecture (Event-Driven + Modular)

The system includes:

  • An event-driven backend that triggers scraping jobs
  • A Python scraping layer (HTTP + parsing)
  • A residential proxy network to emulate real users and support geolocation
  • Storage components:
    • MongoDB for raw product data and schemas
    • A vector database (e.g., Qdrant) for embeddings and fast semantic search
    • Historical price support via storing data over time
  • An optional AI layer (e.g., LangChain + OpenAI) enabling natural-language querying
    • Example: “What is the price of an iPad?” → agent queries the vector DB / data and returns results

Demo / Product Features

The showcased product/UI includes:

  • A form where users enter:
    • ASIN
    • Target countries (e.g., France, Spain, Australia, Japan, UAE, Canada, US, UK, etc.)
  • A price comparison tab with charts/details across countries
  • An AI chat feature that avoids writing SQL or custom parsing—answers are generated by querying stored data

Monitoring and Observability (via Ingest)

Integrated observability shows task lifecycle events such as:

  • “scraping product”
  • invalid/empty product handling
  • retries after failures (timeouts/nonexistent products)
  • embedding step timing in the vector DB
  • finalization that returns results to the UI

It also supports:

  • re-triggering and canceling runs
  • inspecting function calls

Tech Stack (Tools and Roles)

  • Streamlit: front-end UI
  • FastAPI: API layer for request/response handling
  • Ingest: orchestration + logging/monitoring for event-driven workflows
  • Beautiful Soup (with lxml): parse HTML and extract product fields
  • Thor Data: residential proxy provider (core enabler)
    • Emulates real device/browser behavior
    • Supports geolocation and session management
    • Scale mentioned: 60M+ proxy IPs across 190 countries
  • MongoDB (Dockerized): stores scraped product data and schema/model
  • Qdrant: vector database storing embeddings for efficient AI retrieval
  • OpenAI: generates embeddings (and powers LLM responses)
  • LangChain: agent framework connecting tools (scraper, vector DB search, MongoDB retrieval) into an AI workflow

Core Scraping Approach

Instead of headless browser scraping, the approach is:

  • Send requests through Thor Data proxy
  • Retrieve geolocation-specific HTML
  • Parse with lxml + Beautiful Soup

Key extraction pattern

  • Determine the correct Amazon domain by country code
  • Build the product URL from the ASIN
  • Parse DOM tags/attributes:
    • title via structured HTML tags (example: span with product-title property)
    • price via matching A-price structures using regex/text matching
    • brand and other attributes similarly

Notes on complexity

The video mentions an approach where an LLM can help generate scraping code from stored HTML (initially to locate stable tags), but the final shown implementation is tag-based parsing.

AI Querying Design (Retrieval + Structured Results)

When the user asks a question:

  • LangChain agent searches Qdrant using embeddings
  • It then fetches full details from MongoDB
  • OpenAI/LLM generates the response using the retrieved context/tool outputs

This design supports flexible questions without requiring custom queries.

Architecture Flow: Scrape vs. Query

Scrape flow

  1. UI request → Ingest
  2. Ingest triggers FastAPI / scraping function
  3. Scraper calls Thor Data → receives geolocated HTML
  4. Parse with lxml + Beautiful Soup
  5. Store product data in MongoDB
  6. Create embeddings → store in Qdrant
  7. Return results to UI with logs/telemetry

Query flow

  1. UI asks question → Ingest
  2. LangChain agent uses tools:
    • Vector search in Qdrant
    • Retrieve full schema/details from MongoDB
    • Call OpenAI to generate the answer
  3. Return response to the UI

Deployment / Operational Notes

  • Uses Docker Compose to run multiple services together:
    • MongoDB
    • Qdrant
    • Streamlit
    • FastAPI
    • Ingest
  • Includes a README with:
    • run/teardown commands
    • port mappings
    • environment variables
  • Mentions helper “tools” that wrap service operations (vector search, DB lookups, scraping)

Main Speakers / Sources (as stated)

  • Speaker/author: The video creator (unnamed in subtitles), presenting the architecture and demo
  • Sponsored/technical source: Thor Data (residential proxy network), explicitly thanked/sponsored

Original video