RAG at 10 Million Documents — System Design

Summary of technological concepts & system design (RAG at 10M documents)

The video outlines a production-grade Retrieval-Augmented Generation (RAG) system designed to work at ~10 million documents, emphasizing why scale breaks “tutorial RAG” and what architectural components prevent failure modes.

---

Core framing: why “RAG at 1,000 docs” fails at “10 million docs”

Auto-tutorial style RAG can be assembled quickly (vector store + retriever), but at 10M scale the system must handle:

• Messy, duplicate, and inconsistent documents
• Strict security constraints
• Tight latency/speed limits
• Adversarial attacks (e.g., prompt injection)

It’s described as moving from a “lemonade stand” to “air traffic control,” where each system component addresses a specific painful scale-only failure mode.

---

Neighborhood #1: Ingestion (parsing → structure → chunking → metadata)

Goal

Create clean, structured, consistently typed representations of arbitrary documents, then chunk them in a way that preserves meaning.

Key tools/features mentioned

• Apache Tika

  • Acts as a “universal translator” for many file formats.
  • Converts “literally any file” into consistent extracted text + metadata in one pass.
  • Limitation: it provides words/text but not semantic “shape.”

• Unstructured

  • Adds document structure by partitioning into typed elements (e.g., title, narrative text, list items, table).
  • Helps preserve context like headings vs. footnotes.

• Docling

  • Specialized for PDF/layout understanding:
    • multi-column reading order
    • table structure recovery
    • OCR for scanned documents

Chunking (where many RAG systems fail)

• Warning against naive chunking like “split every N characters.”
• Failure mode: cutting tables mid-row creates fragments that embeddings/LLMs can’t interpret (e.g., missing headers).

Recommended approaches:

• Treat tables as atomic units (don’t split them).
• Optionally serialize tables to Markdown to preserve structure as text.
• For running text:
  • Heading detector: each chunk retains the section breadcrumb.
  • Boundary detector: chunk boundaries only at natural sentence/paragraph breaks.

Rule of thumb: Chunk size is a metric; semantic completeness matters most.

Metadata enrichment (precompute retrieval aids)

At scale, metadata becomes essential for filtering and relevance control. The content mentions LlamaIndex metadata extractors that can pre-compute:

• summaries
• keywords
• hypothetical questions

It’s described as “reverse engineering retrieval in advance.”

Glue layer

• LangChain or LlamaIndex are referenced as the orchestration “glue” connecting parser/chunker/extractor steps into a pipeline.

---

Neighborhood #2: Database layer (vector + keyword + SQL filters + re-ranking)

Main points

The beginner assumption (“just use a vector DB”) breaks at 10M due to:

• Vector search being approximate (not exact)
• Semantic vectors failing on exact token matching (IDs, codes, usernames, part numbers)

Vector search internals

• Mentions vector DBs like Pinecone, Weaviate, PGVector
• Notes they use HNSW (Hierarchical Navigable Small World Graphs) to approximate nearest neighbors instead of full scans.
• Trade-off: slight loss of perfect accuracy in exchange for speed.

Limitation: vectors aren’t good for exact matches

Example failure:

• Querying “Stripe error code 402” may return semantically similar “payment failures” but not the exact code.

Solution: hybrid retrieval + re-ranking funnel

A production approach runs multiple retrieval stages:

1. Hybrid search

   • Dense vectors (meaning) + BM25 (exact keyword matching)
   • Mentions Elasticsearch/OpenSearch can fuse both result sets.

2. SQL/relational filtering for hard constraints

   • Reason: vectors aren’t good at strict access control and other boolean constraints.
   • SQL quickly narrows candidates (e.g., date range, public/private tags, allowed user scope).

3. Re-ranking

   • Use a heavier cross-encoder (example: Cohere re-rank)
   • Re-rank only the top ~100 candidates (not all 10M).
   • Benefit: catches subtle relevance gaps that embeddings miss.

Explicit “funnel” rule: Retrieval at scale should not be a single lookup: SQL filters what you’re allowed to see → Hybrid finds likely relevant docs → Re-ranking decides what is truly relevant.

---

Neighborhood #3: Routing, orchestration, safety (agentic control loop)

Orchestrator “brain”

The system moves beyond “search and answer” into agent systems.

Planner + tool execution

• A planner breaks multi-step requests into a checklist (e.g., summarize data and email it).
• An execution mechanism calls tools (calculator, API, email sender).

Conditional router (triage)

• Purpose: avoid running the full retrieval pipeline for every message.
• Router asks: “Does this even need database lookup?”
  • Example: math questions go to a calculator, not retrieval.
• References LlamaIndex router query engine using a fast classifier.

Multi-agent system (parallel specialists)

• When one query is too big, split work among agents:
  • research agent
  • analysis agent
  • risk-flagging agent
• Uses orchestration frameworks like LangGraph or CrewAI to manage:
  • who talks to whom
  • ordering
  • how outputs merge

Feedback loop / self-correction

• If confidence is low:
  • retry with different retrieval strategies
  • route differently
  • sometimes use a different agent/path

Human-in-the-loop safety trade-offs

• Full autonomy is acceptable for cheap/reversible actions.
• Expensive/irreversible actions require human validation:
  • wiring money
  • deleting records
  • sending legal commitments
• Mentions an auditor with an immutable trail for compliance.

Adversarial robustness & red teaming

Attack types and defenses:

• Prompt injection
  • malicious instructions inside documents to override the system prompt
• Information evasion
  • phrasing questions to bypass security filters
• Biased opinions
  • testing whether legacy content causes the system to parrot harmful/bias-laden narratives

Defense examples:

• Red teaming tools: Garak, Lakera, PyRIT
• Guardrails: Nemo Guardrails

These tools continuously attack/stress-test before real attackers do.

---

4) Evaluation (“report card” for everything built)

LLM-based grading

• Uses strong LLM judges to grade weaker pipeline outputs on:
  • faithfulness (did the answer stick to retrieved sources?)
  • relevance

It also references precision/recall for retrieval quality:

• did it retrieve the right chunks?
• did it retrieve enough?

Latency and cost monitoring

• Emphasizes that products can’t be “perfect but slow/expensive.”
• Monitors latency and per-query cost.

Automated evaluation frameworks

• Ragas
• TruLens
• Deep Eval

These are described as continuously scoring performance (not only at launch).

---

Main speakers/sources (as stated)

• Speakers: Not explicitly named in the subtitles.
• Systems/tools referenced (sources within content): Apache Tika, Unstructured, Docling, Elasticsearch, Solr, LlamaIndex, LangChain, Pinecone, Weaviate, PGVector, Elasticsearch/OpenSearch, Cohere re-rank, LangGraph, CrewAI, Garak, Lakera, PyRIT, Nemo Guardrails, Ragas, TruLens, Deep Eval.