Video summary

Online Class LLM on Production – Sesi 1

Main summary

Key takeaways

Educational

Main ideas / lessons conveyed

  • Class structure (7 meetings) and learning progression

    • Meeting 1 is intentionally light/easy (“warm up”).
    • Meetings 2–3 introduce slightly more complex material.
    • Meetings 4–7 cover very complex topics.

  • Incentives / registration note

    • Early registrants receive a limited free C FPI / F benefit (limited by time/quantity; not unlimited).
    • A “second batch” benefit is also mentioned, and it is limited.

  • Two tracks for class tooling & projects

    • The course is split into two tracks:
      • Managed-service track (using APIs)
      • Open-source track (building/fine-tuning/deploying your own models)

  • Planned weekly topics (by meeting number)

    • 2–3: Build using managed service (e.g., via an LLM API).
    • 4: Start fine-tuning an open-source LLM.
    • 5: Learn deploying the LLM and optimization methods.
    • 6: Learn offline vs online evaluation/metrics and build a dashboard to monitor model performance.
    • 7 (last): Make RAG using open-source LLMs.

  • Core framework concept introduced: LLM → LLM use cases

    • The instructor explains what a language model (LM) is and how older approaches evolved into modern LLMs.

Methodology / instructional content (detailed bullets)

A) Required tools / setup advice

  • Install Docker first

    • Docker is emphasized as key to making the class deployment easier.
    • The instructor claims related Docker material exists on their YouTube.

  • Use specific tools/libraries mentioned for the stack

    • Vector database: Pinecone is mentioned ambiguously; later “we will use p vector” and the name “p vector” appear (likely Pinecone).
    • Programming: Python (“py” / “pyon” as misheard “Python”).
    • API framework: “fastapi” is referenced (described as more comprehensive than the instructor’s earlier FastAPI-only approach).
    • Serving models: “CPP” / “all-in-one” style wording appears; later “CPP” and “mem” are mentioned (likely deployment/runtime tools).
    • Performance evaluation tool: “gcard” is mentioned for evaluating performance (details deferred to meeting 6).

  • Colab vs local setup

    • Meetings 1–3: can use a regular laptop + VS Code.
    • Meetings 4+: Colab (Google Colab) is suggested, especially for Mac/Anaconda users due to issues.
    • Mentions that Mac M-series CPUs are “safe” for meetings 1–3.

B) Technical foundations explained (conceptual “how it works”)

  • Language model basics

    • An LM is framed as a probabilistic model that predicts the next token/word (next output).
    • Historical examples mentioned:
      • N-grams (unigram, bigram, trigram)
      • TF-IDF + Naive Bayes (older classic method)
      • Then deep learning approaches: RNN/LSTM/MLP/Transformer

  • Deep learning concepts

    • Deep learning is described as using artificial neural networks.
    • Contrast:
      • Linear function: simple weighted sum
      • Non-linear function: neural nets using activation functions and complex compositions

  • Activation functions overview (examples)

    • Sigmoid: maps values into ~0–1; used for binary classification.
    • Tanh: maps into -1 to 1.
    • ReLU: outputs 0 if negative; otherwise returns the input.
    • Activation functions are emphasized as a transformation of values.

C) LLM definition and evolution (Transformer)

  • What “large” means

    • “Large” refers to very large parameters/weights (hundreds of millions to billions).
    • Cost and accessibility are described as evolving over time (big companies first, then broader availability).

  • Why Transformer matters

    • Transformer is highlighted as the key architecture behind translation and modern GPT-like models.
    • Mentions encoder/decoder separation conceptually.
    • Claims:
      • GPT 1–3 uses Transformer decoder only (per the talk).
    • Notes that training/data preparation and bias avoidance may be more “interesting” than architecture details.

D) Use cases taught/previewed

  • Main use cases for LLMs

    • Problems requiring high language understanding.

  • RAG (Retrieval-Augmented Generation)

    • Described as querying knowledge with user text and retrieving relevant documents.
    • Conceptual pipeline:
      • Prepare documents
      • Store them in a vector database
      • Use the user query to retrieve relevant information
      • Generate grounded answers using the retrieved context
    • Example query given:
      • “effects of implementing object detection in manufacturing”
      • Retrieval returns documents and/or conclusions.

  • Information extraction (entity/document extraction)

    • Extract structured info from unstructured text into JSON (actors, chronology, times, etc.).
    • Framed as useful for product logic and reporting (including CV/resume extraction).

  • Text classification / auto-annotation

    • LLM classifies news into categories (sports, disasters, economics, politics).
    • LLM as an auto-annotator / dataset generator:
      • May require prompt tuning.
      • Warning: purely generating data can create low-quality “non-real” patterns; more acceptable for auto-annotation workflows.

E) Cost guidance / decision heuristic (managed vs open source)

  • Rule-of-thumb heuristic

    • If traffic/requests are small (e.g., 1,000–2,000 range), managed services may be cheaper.
    • If traffic is very large (millions/hundreds of millions), open source may be cheaper overall.

  • Trade-off described

    • Managed services: lower initialization cost but pay per usage.
    • Open-source: higher setup/initial cost but potentially lower per-request cost at scale.

F) Simple guidance on when NOT to use LLM

  • For tasks like fixing address typos, the instructor suggests:
    • Use string similarity / vector similarity methods (e.g., embedding similarity / similarity search).
    • LLM can be overkill (too expensive for that narrow need).

Big assignments / deliverables (what students must produce)

  • Two “big project” assignments

    1. Project 1: RAG using managed service (prototype only)
      • Positioned as a potential portfolio project and a GenAI engineer use-case.
    2. Project 2: RAG using open-source models

  • Evaluation/presentation

    • Projects will be shared in the third meeting (details deferred).
    • Deliverables are posted on LinkedIn.
    • The instructor says only the person who completes both big tasks gets a certificate.

  • Grading approach

    • Perfection is not required:
      • “It doesn’t have to be perfect—coding with errors and learning is okay.”
    • Feedback is provided via personal WhatsApp.
    • Students must follow what the instructor specifies later (details in meeting 3).

Other notable logistics / Q&A themes

  • Mentor involvement

    • Instructor says future sessions will involve mentor teaching and that mentors will be trained.

  • Mac/CPU troubleshooting

    • Acknowledges Mac user issues (especially beyond meetings 1–3).

  • Class positioning

    • Instructor claims this is the only class they are currently teaching.

  • Encouragement

    • Repeats that complex topics become learnable by breaking them into simple components.

Speakers / sources featured

  • Primary speaker: The instructor/host (a single person; identity not provided in the subtitles).

  • Other sources/tools referenced (not speakers)

    • OpenAI GPT (used via managed service)
    • Hugging Face (mentioned for training / datasets)
    • Google (Transformer context and Google search / potential TF-IDF references)
    • Transformer architecture (credit to general “Google research team” / Google’s work mentioned)
    • RAG / vector database concepts (tools referenced include “P vector” / likely Pinecone)
    • Docker, FastAPI, Google Colab, LinkedIn, Python, VS Code
    • TF-IDF, Naive Bayes, RNN/LSTM/MLP, Sigmoid/Tanh/ReLU (as conceptual references)

Original video