Video summary

open-rag-eval: RAG Evaluation without "golden" answers — Ofer Mendelevitch, Vectara

Main summary

Key takeaways

Technology

Summary (RAG Evaluation without “golden” answers)

The video introduces Open RAG Eval, an open-source project from Vectara designed to make RAG evaluation scalable by removing the need for “golden” answers or golden chunks—which are often expensive and difficult to scale.

Core problem addressed

  • Traditional RAG evaluation frequently relies on golden datasets (golden answers/chunks).
  • Open RAG Eval proposes research-backed metrics that work without golden references.

How the system works (architecture)

  1. Input queries

    • Provide a set of queries (e.g., 10–1,000) relevant to the RAG system.

  2. RAG connectors

    • Connect to existing pipelines (examples mentioned):
      • Vectara
      • LangChain
      • LlamaIndex
      • plus other connectors “growing” over time
    • Collects:
      • retrieved chunks
      • generated answers
      • other outputs needed for evaluation

  3. Evaluation step

    • Runs multiple metrics, organized into evaluators.
    • Produces evaluation files containing results for assessment.

  4. Optional UI for analysis

    • The evaluation files can be drag-and-dropped into openevaluation.ai (shown in the talk).
    • The UI helps compare retrieval and generation scores across queries and systems.

Key metrics (no golden answers/chunks)

1) Umbrella (retrieval metric; no golden chunks)

  • Scores a retrieved chunk/passage on a 0–3 scale:
    • 0: irrelevant to the query
    • 3: dedicated to the query and contains the exact answer
  • Uses research from the University of Waterloo / Jimmy Lin lab, reporting correlation with human judgment even without golden chunks.

2) AutoNuggetizer (generation metric; no golden answers)

  • Produces evaluation units called nuggets, via:
    1. Nugget creation (atomic units)
    2. Assign labels/ratings to nuggets (described as “vital or okay rating”)
    3. Select top nuggets (top 20 mentioned)
    4. Use an LLM judge to verify whether the response fully supports or partially supports each selected nugget
  • Intended to evaluate generation quality without needing golden answers.

3) Citation faithfulness

  • Assesses whether citations in the response match the cited content, using support levels such as:
    • fully supported
    • partially supported
    • no support for the cited passage

4) Hallucination detection (Vectara model)

  • Uses Vectara’s hallucination detection model (described as HHM) to check whether the entire response aligns with the retrieved content.

Product/community features

  • Includes connectors for Vectara, LangChain, LlamaIndex, and encourages adding more through issues/PRs.
  • Emphasizes transparency: metrics are research-backed and the implementation is open-source.

Main speaker / source

  • Ofer Mendelevitch (Vectara) — developer relations lead
  • Project research credited to University of Waterloo, Jimmy Lin lab

Original video