Video summary

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Main summary

Key takeaways

Technology

Summary of technological concepts, product features, and lessons (RAG stack “after 37 fails”)

Jonathan Fernandes shares the RAG (Retrieval-Augmented Generation) architecture he converged on after repeated failures, emphasizing a “most ROI per minute” guide. He breaks the stack into standard RAG components and then demonstrates an end-to-end prototype in Google Colab, followed by guidance for production-grade deployments (often on-prem and Docker-based, especially for financial institutions).

1) Target architecture components (the “RAG stack”)

He decomposes the RAG system into these components:

  • Orchestration layer (how pipeline steps are wired)
  • Embedding models (convert text + queries into vectors)
  • Vector database (store + semantic search over embeddings)
  • Large language model (LLM) (generate the final answer using retrieved context)
  • Re-ranking / post-retrieval accuracy improvement
  • Monitoring, tracing, and evaluation (to debug and measure quality)

He frames the approach as:

  • Prototyping using Google Colab
  • Production using Docker (to support on-prem processing and controlled data handling)

2) Recommended tooling choices by stage

Orchestration

  • Prototyping: LlamaIndex or LangGraph
  • Production: LlamaIndex

Embeddings

  • Closed/API models for simplicity during early phases
  • Open embedding models for production flexibility

Examples mentioned:

  • Closed: OpenAI embeddings (e.g., text-embedding-ada-002, text-embedding-3-large)
  • Open: BGE models (e.g., “BGE small”) via Hugging Face

Vector database

  • Qdrant
    • Highlighted as scaling well from a few documents up to hundreds of thousands.

LLMs

  • Closed models preferred for simplicity (API-driven)
    • Example: GPT-3.5 Turbo initially, then GPT-4
  • Open LLM serving for production
    • Ollama or Hugging Face Text Generation Inference (TGI) mentioned
    • Examples referenced: Llama 3.x and Qwen models served in a Docker-friendly setup

Monitoring / tracing

  • Langfuse or LangSmith / Arize Phoenix (naming varies in subtitles, but both are presented as solutions)
    • Prototype: Langfuse / Phoenix
    • Production: Arize Phoenix emphasized (easy in Docker)

Re-ranking (accuracy improvement)

  • Use a cross-encoder (reranker) after retrieval
    • Re-ranker compares query+document more directly for better relevance
  • Examples:
    • Closed reranker: Cohere
    • Open reranker: Nvidia option (in the Dockerized production environment)

Evaluation

  • Use RAGAS to evaluate RAG quality (and do it “painlessly” by leveraging the LLM)
  • RAGAS is presented as a framework to score and analyze quality across more than one query.

3) Tutorial/demonstration flow in Google Colab (knowledge base example)

He uses a small HTML knowledge base for a transit/rail scenario in London (question example: “Where can I get help in London?”).

He describes the RAG pipeline stages:

  1. Retrieval: semantic search in the vector DB using embeddings
  2. Augmentation: combine the retrieved text with the original query as context
  3. Generation: the LLM answers using that context

He then shows a naive RAG implementation with LlamaIndex:

  • Read HTML files from a directory
  • Store embeddings in an in-memory vector store initially
  • Query via the embedding similarity search

Result: the answer is not satisfactory (poor responses attributed to retrieval/embedding/LLM configuration issues).

4) “Fail → improve” sequence using specific upgrades

Improvement 1: change models (embedding + LLM)

He modifies:

  • Embeddings: from an OpenAI embedding to an open BGE embedding model (downloaded into Colab)
  • LLM: from GPT-3.5 Turbo to GPT-4 (temperature set to 0)

Outcome: the answer changes, but still remains not fully helpful.

He inspects source nodes (LlamaIndex feature) to see which HTML files were retrieved and fed into the prompt—useful for debugging relevance.

Improvement 2: add query post-processing / cleaning (concept)

He notes that query processing should include:

  • Removing PII if present (before passing to RAG components)

Improvement 3: add a re-ranker (cross-encoder)

He improves accuracy by:

  • Using retrieval from Qdrant to get top candidates
  • Applying a cross-encoder reranker (specifically a Cohere reranker)
  • Re-ranking the top 2 results

Outcome: the response becomes best so far and is anchored to a more specific location (example: London St. Pancras International, “booth number five…”).

5) Key technical explanation: cross-encoder vs bi-encoder

He explains why reranking helps and how to scale:

  • Cross-encoder
    • Takes query and document together (both into BERT + classifier)
    • Produces a similarity score between 0 and 1
    • More accurate but slow / not scalable for large candidate sets
  • Bi-encoder
    • Encodes query and document separately, then compares via cosine similarity
    • Fast and scalable for retrieval in vector DB

Where they fit in the RAG pipeline:

  • Bi-encoder stage: used for retrieval (vector search)
  • Cross-encoder / reranker stage: used post-retrieval on a small candidate set for accuracy

6) Production deployment pattern (Docker Compose + containers)

He sketches a production environment using Docker Compose, including containers such as:

  • Ingestor: loads HTML files into the system / knowledge base
  • Qdrant: vector DB service
  • Front-end app
  • Model serving:
    • Ollama or Hugging Face TGI
  • Tracing/monitoring: Arize Phoenix
  • Evaluation: RAGAS

He also references a comp.yaml configuration and Docker images pulling from registries (e.g., Qdrant from Docker Hub).

7) Overall “lessons” emphasized

  • Use an ROI-focused stack:
    • Fast prototyping (Colab)
    • Production readiness (Docker, on-prem)
  • Qdrant for scalable vector storage
  • Bi-encoder retrieval + cross-encoder reranking for best relevance
  • Monitoring/tracing is critical for troubleshooting time spent in each component
  • RAGAS evaluation is necessary to validate quality across many queries, not just one

Main speakers/sources

  • Speaker: Jonathan Fernandes (independent AI engineer)
  • Tools/systems referenced as sources: LlamaIndex, LangGraph, Qdrant, Arize Phoenix / Langfuse/LangSmith (monitoring/tracing), Cohere (reranker), Nvidia (open embedding/reranking examples), OpenAI (embeddings/LLMs), Ollama, Hugging Face TGI, RAGAS, BGE (embeddings via Hugging Face), Docker/Docker Compose.

Original video