Learn RAG From Scratch – Python AI Tutorial from a LangChain Engineer

High-level summary

This is a multi-part tutorial by Lance Martin (software engineer at LangChain) teaching Retrieval-Augmented Generation (RAG) end-to-end, from concepts to production patterns. The course mixes conceptual slides, paper references, and runnable LangChain notebooks (indexing → retrieval → generation), with deep dives on query rewriting, routing, query construction, indexing strategies, and active/adaptive RAG flows.

Core RAG pipeline and building blocks

Three canonical stages:

1. Indexing: prepare external documents so they can be searched (split, embed, store in a vector store or other DB).
2. Retrieval: embed the query, use similarity search (k-NN) or other techniques to fetch relevant chunks.
3. Generation: insert retrieved context into a prompt template and run an LLM to produce grounded answers.

Practical code patterns (LangChain examples):

• Create a retriever, split documents, compute embeddings (example: OpenAI 1536-d vectors), store in a vector DB (Chroma demo).
• Tune retrieval parameter K to control number of neighbors.
• Use prompt templates with keys like {context} and {question}; combine retriever + prompt + LLM via a chain (invoke/stream).
• Use LangSmith for run tracing and debugging.

Indexing techniques (how to represent documents)

• Sparse vs dense vectors:

  • Sparse (bag-of-words / TF-like) methods historically used for keyword search.
  • Dense embeddings (learned models) are now commonly used.

• Multi-representation / proposition indexing:

  • Generate a retrieval-optimized representation (summary/proposition) for each document, embed that for retrieval, and store the full raw document separately. Improves recall while preserving full context for generation.

• Hierarchical indexing (Raptor):

  • Cluster documents and summarize clusters recursively to build an abstraction hierarchy. Index both leaf-level chunks and higher-level summaries so retrieval can match low- or high-level questions.

• Token-level / late-interaction retrieval (ColBERT-style):

  • Compute token-level vectors and score via tokenwise max/sum similarity instead of a single vector per document. Offers strong retrieval performance for some tasks but brings latency and production complexity trade-offs.

• HyDE (Hypothetical Document Embeddings):

  • Generate a hypothetical document from the query (via an LLM) and search using that embedding—useful when short user queries are poor retrieval anchors.

Retrieval strategies and post-retrieval processing

• k-NN similarity neighborhoods in embedding space; tune K for task needs.
• Reranking and filtering after retrieval (use rankers to grade relevance).
• Reciprocal Rank Fusion (RRF) aggregation:
  • When expanding a query into many rewrites (multi-query), retrieve for each and fuse ranked lists (RAG Fusion) to consolidate the best candidates.
• Multi-query and query rewriting:
  • Fan-out a user question into multiple rephrasings to increase recall; combine with RRF for aggregation.
• Decomposition (least-to-most / multi-step):
  • Break a complex question into subquestions, retrieve & answer sequentially (answers feed later subquestions), possibly interleaving retrieval with chain-of-thought reasoning.
• Step-back prompting:
  • Create a higher-level “step-back” question (with few-shot examples) that better aligns with concept-level documents; retrieve on both original and step-back queries.
• HyDE / hypothetical-document rewriting (see above).

Query construction and routing

• Query construction (text → DB DSL):

  • Convert natural-language queries into domain-specific query objects (metadata filters for vector stores, text→SQL, text→Cypher).
  • Pattern: send schema/field definitions to the LLM (function-calling / structured outputs) to get a parsed query object (Pydantic-style), then execute.

• Routing:

  • Logical routing: describe available data sources to the LLM and have it output the chosen source as a structured value (function-calling / schema) to route the question.
  • Semantic routing: embed candidate prompts or source descriptors and pick the most similar via embedding similarity.
  • Targets for routing: vector stores, SQL, graph DB, web search, or an LLM fallback.

Active / adaptive RAG and flow engineering

• Active/adaptive RAG: add runtime checks and corrective loops rather than single-shot RAG:

  • Grade retrieved documents for relevance; if none are sufficient, rewrite the query or fall back to web search.
  • Grade generated answers for hallucinations or completeness; if failing, re-retrieve or regenerate.

• Self-RAG / Corrective RAG / CAG / Adaptive RAG:

  • Approaches that incorporate grading, query refinement, and fallback web search into the RAG loop.

• LangGraph / flow-as-state-machine:

  • Model flows as a state-machine/graph with nodes (retrieve, grade, web-search, generate, grade-answer) and conditional edges to implement active RAG. Nodes read/modify shared state for auditable, repeatable flows.

• Practical trade-offs:

  • Use small, fast models for grading/routing to reduce latency (examples: Cohere’s Command R).
  • Use LangSmith + LangGraph for tracing and inspecting nodes/edges to simplify debugging and auditing.

Query translation techniques covered

• Multi-query: generate multiple rephrasings → retrieve each → union
• RAG Fusion: multi-query + reciprocal rank fusion
• Question decomposition: least-to-most, sequential subquestions
• Step-back prompting: generate a higher-level question
• HyDE/Hyde: generate a hypothetical document from the query and retrieve on it

Note: “HyDE” was the term used in the literature and referenced in the demos.

Query construction examples

• Text → metadata filters for vector stores (e.g., publish_date range, views range, title/content semantic search).
• Use function-calling / structured output (Pydantic-like schemas) to reliably translate NL into structured query objects.

Indexing product integrations & libraries mentioned

• LangChain (chains, retrievers, expression language)
• LangGraph (graph/state-machine flows)
• LangSmith (tracing/observability)
• Vector stores: Chroma (demo)
• Embedding providers: OpenAI embeddings, Cohere embeddings (Command R)
• Function-calling / structured outputs (OpenAI-like function schema) used for routing, query construction, grading
• Specialized retrieval/indexing libraries referenced: Raptor, ColBERT, etc.

Empirical analysis: “Is RAG dead?” and long-context LLMs

• Context windows have grown dramatically (≈8k → 100k → 1M tokens), raising the question of RAG’s relevance.
• Needle-in-haystack experiments (multi-needle extension) show limits:
  • Retrieval and reasoning degrade as the number of facts increases.
  • Recency bias: earlier-in-context facts are harder to retrieve than later ones.
  • Long contexts do not guarantee robust multi-fact retrieval + reasoning; RAG and pre-/post-retrieval reasoning remain valuable.
• Conclusion: RAG will evolve rather than disappear—document-centric approaches, adaptive flows, and new indexing strategies remain important.

Practical recommendations and trade-offs

• Favor document-centric retrieval (full-document summaries or multi-representation) over brittle chunking when possible.
• Use hierarchical (Raptor) or multi-representation indexing to support both high-level and low-level questions.
• Add runtime checks (relevance, hallucination) and fallback strategies (web search) for production resilience.
• Use structured outputs / function-calling to produce deterministic routing and query objects (reduces brittleness).
• For grading and routing, prefer smaller/faster models to lower latency; for final generation, use larger models as required.

Tutorials, demos & notebooks referenced

Notebooks and demos include:

• Basic RAG end-to-end: indexing → retrieval → generation.
• Indexing deep dive: embeddings, splitting, Chroma + OpenAI embeddings.
• Retrieval deep dive: k-NN similarity and K tuning.
• Generation: prompt templates and chains.
• Multi-query demo: generate queries → retrieve → union.
• RAG Fusion demo: reciprocal rank fusion aggregation.
• Decomposition demo: break questions into subquestions; interleaved retrieval + chaining.
• Step-back prompting demo.
• HyDE (hypothetical doc generation) demo.
• Routing demo: logical (structured outputs/function-calling) and semantic (embeddings).
• Query construction demo: NL → metadata filters; text→SQL/Cypher patterns.
• Multi-representation indexing demo: summaries in vector store + raw docs stored separately.
• Raptor hierarchical indexing demo: cluster & summarize recursively.
• ColBERT-style late interaction demo: token-level retrieval.
• LangGraph active/adaptive RAG demo: state-machine with graders and web-search fallback.
• Adaptive RAG / Corrective RAG example: routing, grading, fallbacks.
• “Is RAG dead?” analysis and needle-in-haystack experiments.

Papers, methods and keywords to consult

• Retrieval-Augmented Generation (RAG)
• RAG Fusion / Reciprocal Rank Fusion (RRF)
• HyDE (Hypothetical Document Embeddings)
• Proposition / DenseX / multi-representation indexing
• Raptor (hierarchical summarization / indexing)
• ColBERT (token-level / late-interaction retrieval)
• Least-to-most prompting (question decomposition)
• Step-back prompting (Google)
• Self-RAG, Corrective RAG, Adaptive RAG
• Needle-in-haystack analyses (Anthropic + extensions)
• Command R (Cohere model recommended for fast routing/grading in demos)

Main speakers / sources

• Main presenter: Lance Martin — Software Engineer at LangChain (author of the video series).
• Primary technologies: LangChain, LangGraph, LangSmith, Chroma, OpenAI (embeddings & LLMs), Cohere (Command R).
• Other contributors referenced: Greg Cameron (needle analyses) and multiple research papers/authors (Google, Anthropic, Raptor/HyDE/Corrective-RAG authors).

Additional deliverables referenced

• A concise checklist for building a production RAG system (components, tests, budget/latency trade-offs).
• A list of the notebooks/demos in lesson order with quick commands to run them locally.