Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft

Summary

Nick Ung and Akshay Sharma describe how to build evaluation systems that meaningfully predict how user-facing AI agents will perform in production. Their central point is that an evaluation is useful only when it is realistic, tied to specific outcomes, trusted by the team, and connected to a process for fixing failures.

1. Treat evaluation as part of the full agent lifecycle

The speakers frame agent development as two connected stages:

• Before launch: Run offline evaluations to check whether an agent meets defined release criteria. For multi-turn support agents, this means simulating conversations and assessing the resulting interactions.
• After launch: Trace real production interactions, evaluate performance, and use human review to identify failure modes and guide improvements.

They caution against treating evaluation as a box-checking exercise. Scores must serve as meaningful launch or development gates, and teams need clear ownership and follow-up when a regression is detected.

2. Make offline simulations reflect real users

A simulated user should be a credible stand-in for production traffic—not simply a convenient, polite LLM. The speakers found that general-purpose models tended to behave like helpful assistants: patient, articulate, and overly cooperative. This made early evaluations too easy and produced misleadingly high scores.

Their approach is to build simulated users around:

• Intent: What the user is trying to accomplish.
• User data or “world state”: Relevant circumstances and background information.
• Persona and behavior: How the user communicates, including frustration, skepticism, impatience, or reluctance to engage.

They also recommend using samples of real production data where appropriate and deliberately covering different expected paths and edge cases. Fine-tuning or otherwise configuring a simulated user to better resemble real users may lower evaluation scores; the speakers argue that this can be a sign of a more realistic and useful test.

3. Combine language-model judges with deterministic checks

An evaluation system can use both:

• LLM judges for qualitative aspects of a conversation.
• Deterministic code checks for conditions that can be tested directly, such as whether the agent made an expected tool call or followed a required action.

Generic metrics—such as relevance, usefulness, naturalness, or completeness—can be useful as a starting point, but the speakers say they are often too vague to guide action. Evaluation criteria should instead reflect concrete task success, product requirements, or business outcomes.

4. Calibrate judges against human judgments

The speakers recommend treating an LLM judge like a classifier whose results must be checked against human labels.

A practical workflow is:

1. Work with subject-matter experts to define clear success and failure criteria for important tasks.
2. Prefer well-defined, often binary outcomes when possible; these are easier to label and calibrate.
3. Collect a set of human-labeled examples—Akshay describes using roughly 100 as a starting point.
4. Divide the examples into sets for prompt development, iteration, and held-out testing.
5. Refine the judge’s instructions using development examples, then check its performance on examples not used for refinement.
6. Compare the judge’s results with human labels using measures such as accuracy and completeness.

The rubric itself should also evolve. As teams review more data, they may discover that the original criteria missed important distinctions. Evaluation criteria and the judge therefore need to be refined together.

5. Report scores with appropriate statistical caution

A score is an estimate, not a guarantee. The speakers recommend considering confidence intervals and sampling quality when reporting results—especially when a score will influence a launch decision or be presented to executives. For example, a small difference between two scores on only a few dozen examples may not be enough to show that one agent is genuinely better.

6. Run a continuous error-analysis loop

Evaluation should be an ongoing process rather than a one-time audit. The speakers outline a recurring loop:

1. Log and inspect agent traces.
2. Identify recurring failure modes.
3. Retain metrics that affect decisions or reflect product and functional requirements; remove noise.
4. Form hypotheses about what to change.
5. Update the agent, evaluation criteria, or both.
6. Rerun evaluations and repeat the process regularly.

Traces should capture the agent’s execution, including the steps taken, model inputs, tool calls, token usage, and latency. Annotation interfaces can let subject-matter experts review interactions without needing to inspect raw trace data. Their labels can then support judge calibration and become examples in offline evaluation datasets.

7. Use evaluation findings to improve the whole system

Not every failure calls for changing model weights. The speakers distinguish several improvement paths:

• Model learning: Train or fine-tune the model.
• Contextual improvements: Improve the information available to the agent, such as documents or user memory.
• Resource and orchestration improvements: Adjust prompts, tool flows, retries, routing, and other control logic.

The goal of evaluation is to identify which kind of change is likely to address a failure.

8. Standardize and scale evaluation

The speakers want evaluation to be repeatable and accessible beyond individual engineers’ scripts and notebooks. They describe a configurable system with reusable elements such as tasks, datasets, personas, model adapters, and evaluators. Configurations can be stored in YAML so analysts and data specialists can contribute.

Once defined, evaluation suites can be run at multiple points in development—for example, locally, before commits, and in CI/CD as regression or acceptance tests. Scaling to large datasets also requires sufficient parallel execution capacity.

Speakers and sources mentioned

Speakers

• Nick Ung — Lyft; introduced the evaluation lifecycle, user simulation, and the work on closing the loop and systematization.
• Akshay Sharma (“Ash” in the subtitles) — Lyft; discussed LLM-judge rubrics, human calibration, statistical precision, tracing, and error analysis.

Sources, tools, and approaches referenced

• A customer-support-agent benchmark from Sierra AI, referred to in the subtitles as “Towbench” (likely τ-bench).
• A Microsoft article or work on user simulation.
• DeepEval prebuilt evaluation metrics.
• LangSmith and Langfuse for tracing and trace review.