How to Systematically Setup LLM Evals (Metrics, Unit Tests, LLM-as-a-Judge)

Main ideas / lessons conveyed

• Agentic/LLM systems fail in subtle, hard-to-predict ways, and many deployments never become reliable without evaluation.
• Evaluation is central to shipping and improving LLM apps, because you need to detect:
  • Prompt/system changes causing new issues
  • Pipeline/classification correctness
  • Style and tone requirements (not just factuality)
  • Safety/unsafe outputs and robustness to prompt injection
  • Performance degradation over time in production (data/prompt/model drift)
• Reported industry stats (potentially inflated/uncertain) are used to motivate urgency:
  • Gartner-style claim: by 2027, 40% of agent AI projects expected to be cancelled
  • MIT-style claim: 95% of generative AI pilots failing
• Continuous improvement must be systematic (not accidental):
  • Measure and debug failures
  • Apply changes guided by evaluation results
  • Maintain an ongoing improvement loop
• The video introduces three levels of LLM evaluations with increasing cost/effort:
  1. Level 1: Fast automated unit tests
  2. Level 2: Human + model evaluations (LLM-as-a-judge with human alignment)
  3. Level 3: A/B testing with real users to measure business impact
• Metric selection matters:
  • Start with simple binary “good/bad” metrics
  • Use either:
    • Reference-based metrics (gold answer exists)
    • Reference-free metrics (multiple answers can be acceptable; judge criteria like tone/safety/format)
• Common mistakes to avoid:
  • “Tool-first thinking” (buying platforms/models instead of defining meaningful evals)
  • “Generic metric obsession” (meaningless dashboards)
  • “Avoiding data” (not reviewing traces/events)
  • “Unaligned LLM judges” (not validating judge behavior against humans)
• A key workflow theme: Analyze → Measure → Improve.
  • Collect real/raw examples and failure modes
  • Convert to metrics
  • Iterate prompts/models/architecture and re-evaluate

---

Evaluation-driven improvement cycle (Analyze → Measure → Improve)

Analyze

• Collect examples flowing through your system (preferably raw events).
• Categorize failure modes from:
  • user complaints
  • observed application errors
  • crashes / missing outputs / storage issues
• Use tools (speaker mentions Langfuse) to inspect traces and data.

Measure

• Turn failure insights into quantitative metrics.
• Use metrics at multiple granularity:
  • boolean pass/fail
  • scaled scores (e.g., 1–0, 1–5, 1–10)
  • rankings / specific scoring dimensions

Improve

• Refine prompts, try different models, and adjust architecture.
• Repeat the cycle as data and prompts drift over time.

---

Three levels of evaluations (when/how often and purpose)

Level 1: Unit tests (fast automated assertions)

• Purpose
  • Catch regressions immediately after code/prompt changes
  • Ensure core functionality and data/format expectations hold
• How to think about unit tests for LLM apps
  • Create assertions around structured output fields and types
  • Store raw events (JSON) to replay tests
• Frequency
  • Run on every significant change (implied: every prompt/code change)
• Cost
  • Low (fast, cheap)
• Example assertion types
  • Category must be one of allowed labels
  • Confidence must be a float in a range (e.g., 0 to 1)
  • Response length must exceed a threshold

Level 2: Human + model evaluations (LLM-as-a-judge, aligned to human criteria)

• Purpose
  • Systematically review quality dimensions that unit tests can’t cover well
  • Provide critiques beyond just pass/fail
• Human-in-the-loop requirement (key rule)
  • Avoid “fully automated model monitoring” at first
  • Ensure the eval aligns with how humans/domain experts judge outputs
• Frequency
  • Suggested as weekly or bi-weekly for production systems (depending on complexity)
• Process to align “LLM-as-a-judge” with humans
  • Collect a dataset of inputs + model outputs
  • Have humans evaluate the same pairs and produce a “gold-ish” standard
  • Use an LLM to critique/rate outputs based on defined criteria
  • Track human vs model agreement
  • Iterate on the judge prompt until agreement improves
  • Continue monitoring because drift occurs over time (prompt/data/user changes)
• Example judge prompt behavior (conceptual structure)
  • Judge criteria (e.g., accuracy, helpfulness, tone)
  • Request detailed critique explaining why
  • Output a binary (good/bad) or boolean verdict via structured output

Level 3: A/B testing (real user experiments)

• Purpose
  • Measure business impact of changes (not just eval scores)
• What to test
  • Compare different prompts, different models, or even different workflows
• When to run
  • With major releases
  • When you want confidence that improvements affect user outcomes
• Cost
  • High (requires meaningful user traffic + data)
• Metrics that can be used
  • user satisfaction
  • task completion rate
  • time to resolution
  • engagement
  • business outcomes (sales, retention)
• Complexity warning
  • Hard to fully automate user-feedback AB testing; may start with small prompt AB tests.

---

Building Level 1 unit tests for an LLM workflow (example implementation pattern)

• Prepare an events folder containing raw JSON tickets
  • These should reflect the exact schema your workflow accepts
  • Validate incoming events with a schema/typing approach (speaker mentions Pydantic)
• Create a workflow that takes each event through:
  • classification (structured output)
  • routing/next step
  • response generation
• Add an evals/unit-tests script that:
  • Loads each JSON event
  • Runs the workflow
  • Uses Python assertions (assert ...) to validate outputs
• Assertion examples given
  • category is in allowed set (e.g., billing/technical/general)
  • confidence is a float within 0–1
  • response length is > N (to ensure it produced something)
  • for a specific test case, expected category equals a target label
• Test organization concept
  • Put tests in an evals/ folder (and events alongside)
  • Loop through multiple event scenarios; allow some to fail intentionally to demonstrate behavior
• Growth guidance
  • Expand assertions across multiple workflow steps as complexity increases (classification → router output → confidence fields → response text → API success flags)
  • As the system grows, store tests/events in a DB with metadata and automate retrieval
  • Consider using a standard testing framework like pytest
• Feedback cycle for local iteration
  • Run tests locally before deploying
  • Make changes, confirm failing tests pass
  • Re-run the whole suite to ensure other cases remain correct

---

Metric types and how to choose them

• Start simple:
  • “good/bad” binary judgments are easiest to implement and validate
• Two metric categories:
  • Reference-based metrics (gold answer known)
    • exact string matching
    • semantic similarity
    • code execution results
    • SQL query correctness
    • structured data validation
  • Reference-free metrics (no single canonical answer)
    • tone appropriateness
    • length constraints
    • hallucination detection
    • format compliance
    • safety/toxicity
• Use 1–0 or 1–5 / 1–10 style scoring only when criteria are clear and aligned with humans.

---

Speakers / sources featured (as identified in the subtitles)

• Dave (host/speaker; “Hey, what’s up? Dave here…”)
• Data Luminna (company referenced as the source of the strategy/tools; not a distinct individual speaker)
• Langfuse (tool/platform referenced)
• Ham el Hussein and Shankar (authors referenced as producing a report/course on “application-centric AI evals” / “application-centric AI fields for engineers and technical PMs”)
• Gartner (source of the agent project cancellation statistic mentioned)
• MIT (source of the generative AI pilot failure statistic mentioned)
• Zapier (anecdote source: principal engineer looking at traces/data)