Agentic Evaluations Workshop - Deep Dive on the Future on Evals for Agents.

Main ideas & lessons from the workshop

• Agentic evaluation (“agentic e-vals”) is harder than LLM text evaluation

  • Agents plan, decide, and act and often operate over long-running, multi-step workflows.
  • Agent evaluation becomes difficult because outcomes can depend on state over time, interaction sequences, and external/environmental dynamics.

• Honest, transparent evaluation reporting is a major missing piece

  • Evaluation reporting often hides key details in “small print,” enabling benchmark gaming and reducing reproducibility.
  • Examples mentioned include omitted subsets of benchmark problems and “benchmark mixing” across different model versions.
  • Reporting issues also include:
    • Inconsistent graphics/misleading charts
    • Lack of error bars
    • Weak practices in social impact eval reporting (often less transparent over time)

• Standardization and centralization of eval data are necessary

  • The workshop emphasizes shared schemas, complete traceability, and centralized indexing so results can be compared fairly and audited.

• Reliability is distinct from capability

  • Even when agents get better at benchmark tasks, usefulness in the real world depends on reliability (how often it works, how consistently, robustness to changes, and how severe failures are).
  • A major claim: capability rises faster than reliability, implying a widening “capability–reliability gap.”

• Dynamic, multi-app simulation is important for realistic agent evaluation

  • Benchmarks must reflect that real environments change (emails arrive, calendars change, prices fluctuate).
  • Gaia 2 is presented as an agent benchmark built around simulated app universes with dynamic events and robust verification.

• Environment-first evaluation in practice

  • A pragmatic methodology: define environments/sandboxes, choose tasks, and implement grading/verifiers before iterating on agents.
  • Evaluations should include safety checks and guard against reward hacking.

• “Living benchmarks” and community maintenance

  • Maintaining leaderboards and eval frameworks is burdensome and causes score fragmentation across publications and time.
  • Proposed solution: Community Eval / living benchmark infrastructure on the Hub:
    • standardized run definitions (e.g., eval.yaml)
    • automated publishing of results
    • community PR-based updates and discussions

---

Methodologies and instruction-like frameworks presented

A) “Every E-val Ever” (centralized eval documentation & schema) — detailed components

Purpose

• Centralize first- and third-party evals so results are not scattered across incompatible formats (leaderboards, blogs, reports).
• Improve honesty and reduce gamification by controlling confounders.

Core artifacts

• A unified open data format/schema
• A public dataset on Hugging Face containing evaluation results collected in that schema
• Eval cards: a UI concept to browse all evals for a model organized by category

Schema capabilities

• Aggregate results support
  • Require reporting of:
    • source provenance
    • model specification
    • quantization
    • model/version
    • evaluation library
    • etc.
• Instance/line-by-line results support
  • Supports detailed results beyond a single aggregate score

Goal

• Enable reproducible comparisons by fixing variables like version, quantization, and evaluation tooling.

---

B) Agentic evaluation requirements (Balaji’s “what’s missing”) — checklist style

• Session-level reporting
  • Report session-level details because agent runs can vary across sessions even with the same nominal score.
  • Improves reproducibility.
• Agent identity / composition
  • Do not treat “model name” as the agent.
  • Report:
    • sub-agent list
    • MCP servers
    • memory configuration
    • system composition details
• Avoid benchmark protocols that block generalists
  • Benchmarks requiring special setup can prevent evaluating general-purpose agents fairly.
• Standardize robustness measurements
  • Common missing items:
    • seeds
    • prompt perturbations
    • pass@K reporting
• Cost reporting consistency
  • Some evals report cost; others omit it—needs standardization.
• Toward cross-agent, cross-protocol, cross-environment, agent-agnostic evaluation
  • The aim is not simultaneously comparing only one slice of the problem.

---

C) Reliability measurement framework (Arvind Narayanan) — dimensions & what to measure

• Define reliability as orthogonal to capability

• Four emergent dimensions

  1. Consistency
     • Outcome consistency: same pass/fail each time vs stochastic success/failure within a task
     • Trajectory consistency: similar action sequences vs variable action paths
     • Cost/resource consistency: stability of resource use across runs
  2. Robustness
     • Fault robustness: inject faults into the environment (e.g., API timeouts)
     • Prompt robustness: rewrite prompts while preserving semantics (style/tone changes)
  3. Predictability / Calibration
     • Can an evaluator inspect an agent trace to determine correctness?
     • Calibration: confidence scores match empirical success rates
     • Discrimination: ability to separate success vs failure when confidence is requested
  4. Severity of failures
     • Measure how bad failures are (minor formatting vs major outcomes like data deletion)

• Composite reliability index

  • Reliability improves gradually over ~18 months in reported findings, even when accuracy improves rapidly.
• Task-dependent caution
  • Consistency metrics may be undesirable for tasks requiring creativity (e.g., poetry).

---

D) Gaia 2 benchmark methodology (Pierre Andre / Meta) — simulation & verification approach

Problem addressed

• Traditional evaluations are static: one prompt → one model output.
• Agent evaluations need dynamic adaptation to changing external conditions.

Simulation approach: multi-app simulation

• Agents operate in simulated universes containing multiple “apps”
• Apps expose tools/APIs; users and environment generate events
• The universe changes due to:
  • agent actions
  • user responses
  • environment events

ARE (Agent Research Environment) concepts

• Apps
  • Maintain state and provide tool/API surfaces
• Universe
  • Simulated environment + initial state (emails, calendar, conversations, etc.)
• Injected events
  • Arrive from user, agent, or environment
• Scenarios
  • Structured tasks: an ordered sequence of expected and possible events

Capabilities evaluated (Gaia 2)

1. Execution (single-turn tool-using task that writes/acts)
2. Search (within the app universe across API surfaces)
3. Adaptability (multi-turn; environment changes after actions)
4. Time (multi-week type tasks simulated faster via time jumps)
5. Ambiguity (agent must ask clarifying questions to avoid wrong actions)

Robustness/noise mechanisms

• Tool failures
• Tool/API signature variation to prevent overfitting
• Environment noise (irrelevant external events)
• Agent-to-agent mode (sub-agents with natural language interface to tools)

Verification design (avoid expensive rubric-only judging)

• Action-level verification
  • Compare expected action event graphs to actual action graphs
  • Hard verifiers: cheaper and more reproducible (algorithmic checks / equality)
  • Soft verifiers: LLM-based checks for content where exact equality isn’t required

Anti-hacking measures

• Strict checkers for result patterns
• Increase noise and vary tool signatures so agents cannot rely on fixed tool behavior.

---

E) Environment-first evaluation workflow (Mahesh / Bespoke Labs) — practical steps

How not to evaluate

• Don’t only judge final output text with no intermediate checks (hard to diagnose failures).
• Don’t start with overly granular micro-level checks too early (focus strategy matters).
• Don’t deploy to production to find issues (avoid the “real users as test set” trap).

Why agent evals are hard

• Stochasticity: runs vary
• World/tool interaction: environment changes and can be nondeterministic/expensive

Evaluation setup levels

1. Verifiable domains
   • Use unit tests / mathematical correctness
   • Watch out for:
     • uneven weighting of unit tests
     • reward hacking (e.g., passing tests without fixing root issue)
2. Non-verifiable outputs
   • Use rubrics + LLM-as-judge to convert qualitative outcomes into scores
   • Score aggregation via weighted rubric outputs

Environment definition

• Environment = sandbox with dependencies, state, tools, and data
• Can be real-world-like or simulated to avoid destructive real actions

Key environment design decisions

• Use one or multiple environments (match target domain)
• Prevent grader/solution access to the agent (avoid reward hacking)
• Grade only what you asked for (if you omit evaluation of one required subtask, agents can game it)
• Add checks/constraints (e.g., fingerprinting unit tests to prevent tampering)

Core metrics

• Success rate
• Pass@K
• Plus operational metrics:
  • efficiency (steps)
  • token count
  • latency
  • cost
• Add safety-specific tasks/penalties (e.g., data deletion, credit card misuse)

Workflow

• Measure success and auxiliary metrics
• Inspect traces
• Do automated failure analysis
• Use results to iterate on agents (potentially with optimization like job algorithms)
• Then deploy once evaluation thresholds are met.

---

F) “Community Eval” / living benchmarks (Nathan / Hugging Face) — operational instructions

Goal

• Enable benchmarks to be:
  • decentralized
  • easy to run
  • continuously updated via community participation

Community Eval features

• Benchmarks created from Hub datasets
• Each dataset has an eval.yaml describing:
  • task name/description
  • evaluation framework
  • specs and solver logic (prompts/scaffolding)
  • scorer logic (often LLM-as-judge)

Why reuse existing evaluation frameworks

• Reduce maintenance burden
• Improve standardization and accessibility
• Separate evaluation logic from benchmark definition

Running evaluations

• Prefer controlled execution environments (e.g., HF Jobs) for reproducibility
• Avoid inference-provider black boxes where model prompting/version details may be unclear

Publishing results dynamically

• Evaluation results are published as:
  • PRs that show on leaderboards/dataset pages
  • include detailed logs (temperature, retries, judge settings, etc.)
  • compute statistics like standard error

Community involvement

• Community members can open PRs/discussions when they disagree
• Model authors can hide disputed scores if needed

Outcome

• Reduced “score fragmentation” and more traceable, living updates over time.

---

Speaker list / sources featured (as stated in subtitles)

• Ben (host/moderator; Hugging Face community education and program organizer)
• Balaji Koch (Hugging Face; technical AI policy researcher; leads e-val e-val; talk on evaluation reporting for agentic AI)
• Arvind Narayanan (Princeton; talk on AI agent reliability)
• Pierre Andre (Meta; talk on Gaia 2)
• Mahesh (Bespoke Labs; co-founder and CEO; talk on environments for agent evaluations)
• Nathan (Hugging Face; engineer; maintainer of LightEval, Open LLM leaderboard, and Community Eval)
• Amish Shrestha Morty (Bespoke Labs; mentioned as an industry evaluation speaker, though not clearly shown in the provided subtitles segment as a full talk)
• A speaker referenced as “Averil” / “Avir” (appears to be part of the moderator exchange; likely another attendee/panelist not fully identified in the provided text)
• Abhishek (panelist; referenced in panel discussion, likely “Abishek”)
• Vijay (referenced in discussion about long-horizon tasks; not introduced as a speaker in the provided segment)

Note: Names are taken from the subtitles; some may be partial or misspelled due to auto-generation.