How to Evaluate an AI Agent Before Deployment: 5 Metrics

Updated October 30, 2026. The earlier version of this page described measurements and a production failure that I cannot document, so they are removed. What replaces them is a small harness I wrote and ran on 30 October 2026 with Python 3.12.3 and pytest 9.2.0. The agent inside it is a 27-line rule-based stub standing in for an LLM, so the numbers show how the metrics behave, not how any model performs. Vendor facts were re-read from their own docs and package pages that day.

Agents break differently from ordinary software. The output is text, it can change from run to run, and there is no obvious assertion to write, so many teams ship on a handful of manual tries. That leaves nothing to compare when a prompt change lands. The fix is a repeatable set of cases with expected tool calls, a few numbers computed from it, and a rule for when a release is blocked. If you are building AI agents for a real workflow, this is the minimum I would put in place before the first user.

What an agent eval is, and the three levels

An agent eval is a test that runs the agent on a fixed input and scores what it did: which tools it called, with which arguments, and whether the task finished. It is not a unit test of a function, because the same prompt can produce different output on different runs, and "correct" is sometimes a matter of degree.

Hamel Husain's guide to evals splits the work into three levels: Level 1 is unit tests (assertions, in the same spirit as pytest tests), Level 2 is human and model evaluation, and Level 3 is A/B testing. He notes that cost rises with each level. This page covers Level 1 in full and the pre-release part of Level 2. Level 3 needs production traffic, so it comes after launch.

Step 1: a golden set you can trust

The golden set is the list of cases the agent is scored on. Each case records an input, the tool calls a correct agent would make (with allowed argument ranges), and any text the reply must contain. Mine are hand-written and live next to the code. Two cases from the file:

{
 "id": "ql-01",
 "input": "We are a 15-person design studio, about $800k revenue, want to automate client onboarding",
 "expected_calls": [{"name": "qualify_lead", "args": {"score": {"range": [70, 100]}, "follow_up": true}}],
 "reply_contains": [], "source": "hand-written", "reviewed": "2026-10-30"
},
{
 "id": "hd-01",
 "input": "This is unacceptable, I want a refund",
 "expected_calls": [{"name": "handoff_to_human", "args": {"reason": "complaint"}}],
 "reply_contains": ["person"], "source": "hand-written", "reviewed": "2026-10-30"
}

The file has 20 cases: 12 lead-qualification cases (ql-), 4 booking requests (bk-) and 4 complaints that must be handed to a person (hd-). The stub agent passes all 20 on its first run, and that tells you the set is easy, not that the agent is good. I wrote the stub and the cases together, so they agree by construction. The cases that find real faults come from two places: production traces of things that went wrong, and a reviewer who did not build the agent.

Before trusting the numbers, check the file itself. A mislabelled case produces a failing test that looks like an agent bug. This validator checks for duplicate ids, missing review dates, unknown tools, impossible score ranges, a follow_up flag that contradicts the range, and the same input labelled two ways:

import sys, collections
from evals import load_golden
TOOLS = {"qualify_lead", "book_meeting", "handoff_to_human"}

def validate(cases):
    errors, seen_ids = [], set()
    by_input = collections.defaultdict(list)
    for c in cases:
        if c["id"] in seen_ids: errors.append(f"{c['id']}: duplicate id")
        seen_ids.add(c["id"])
        by_input[c["input"]].append(c)
        if not c.get("reviewed"): errors.append(f"{c['id']}: no reviewer date")
        for e in c["expected_calls"]:
            if e["name"] not in TOOLS: errors.append(f"{c['id']}: unknown tool {e['name']}")
            r = e["args"].get("score", {}).get("range")
            if r and not 0 <= r[0] <= r[1] <= 100: errors.append(f"{c['id']}: bad range {r}")
            if r and "follow_up" in e["args"] and e["args"]["follow_up"] != (r[0] >= 50):
                errors.append(f"{c['id']}: follow_up={e['args']['follow_up']} contradicts range {r}")
    for text, group in by_input.items():
        if len(group) > 1:
            labels = {str(g["expected_calls"]) for g in group}
            ids = [g["id"] for g in group]
            errors.append(f"{ids}: same input, {'CONFLICTING' if len(labels) > 1 else 'identical'} labels")
    return errors

if __name__ == "__main__":
    errs = validate(load_golden(sys.argv[1] if len(sys.argv) > 1 else "golden_set.json"))
    print("\n".join(errs) or "golden set OK")
    sys.exit(1 if errs else 0)

I added one deliberately conflicting case (ql-13, the ql-01 input labelled as a low score) to a copy of the file. The validator rejects the copy and passes the real file:

$ python validate_golden.py golden_set_bad.json
['ql-01', 'ql-13']: same input, CONFLICTING labels
exit=1
$ python validate_golden.py
golden set OK
exit=0

It also caught a mistake of mine in a first draft: case ql-07 had a score range of 40 to 70 but expected follow_up: true, and the follow-up threshold in the stub is 50, so the label contradicted itself. I changed the range to 50 to 70. A validator cannot tell you a label is right, only that it is consistent. Someone other than the author still has to read the expected answers.

Step 2: the five metrics

Here is the scoring code, exactly as run. score_case compares each tool call the agent made with the expected calls, run_suite times each case, and the functions at the bottom turn the rows into the metrics.

import json, math, time
from pathlib import Path
from agent import run_agent

def args_match(actual: dict, expected: dict) -> bool:
    for k, exp in expected.items():
        val = actual.get(k)
        if isinstance(exp, dict) and "range" in exp:
            lo, hi = exp["range"]
            if val is None or not lo <= val <= hi:
                return False
        elif val != exp:
            return False
    return True

def call_matches(actual: dict, expected: dict) -> bool:
    return actual["name"] == expected["name"] and args_match(actual["arguments"], expected["args"])

def score_case(case: dict, result: dict) -> dict:
    actual, expected = result["tool_calls"], case["expected_calls"]
    good = [a for a in actual if any(call_matches(a, e) for e in expected)]
    found = [e for e in expected if any(call_matches(a, e) for a in actual)]
    reply_ok = all(s in result["reply"] for s in case["reply_contains"])
    return {"id": case["id"], "calls": len(actual), "good_calls": len(good),
            "complete": len(found) == len(expected) and reply_ok}

def run_suite(cases: list, version: str) -> list:
    out = []
    for c in cases:
        t0 = time.perf_counter()
        r = run_agent(c["input"], version)
        s = score_case(c, r)
        s["latency_s"] = time.perf_counter() - t0
        s["input"], s["actual"] = c["input"], r["tool_calls"]
        out.append(s)
    return out

def tool_precision(rows): return sum(r["good_calls"] for r in rows) / sum(r["calls"] for r in rows)
def completion_rate(rows): return sum(r["complete"] for r in rows) / len(rows)

def percentile(values, q):  # nearest-rank
    s = sorted(values)
    return s[max(0, math.ceil(q * len(s)) - 1)]

def regression_rate(prev_rows, new_rows):
    prev_pass = {r["id"] for r in prev_rows if r["complete"]}
    new_pass = {r["id"] for r in new_rows if r["complete"]}
    return len(prev_pass - new_pass) / len(prev_pass), sorted(prev_pass - new_pass)

def load_golden(path="golden_set.json"): return json.loads(Path(path).read_text())

1. Tool call precision

Precision is the share of the agent's tool calls that match an expected call by name and arguments: correct calls divided by all calls made. It drops when the agent calls the wrong tool, passes bad arguments, or adds calls nobody asked for. Recall, the share of expected calls that happened, is the mirror image and is covered by task completion below. An extra or wrong call can update the wrong record or message the wrong person, so for tool-using agents this is the metric to put first.

DeepEval ships a ToolCorrectnessMetric. By default it compares tool names, and you can require input parameters and outputs to match too. I tried it on the complaint case with DeepEval 4.3.2. Two things were worth knowing. Constructing the metric raised an error when no OPENAI_API_KEY was set, even though I was only matching tool names. With a placeholder key and outbound traffic pointed at a closed proxy port, measure() completed, which shows name matching did not need a judge call:

from deepeval.test_case import LLMTestCase, ToolCall
from deepeval.metrics import ToolCorrectnessMetric
from agent import run_agent

text = "This is unacceptable, I want a refund"
for v in ("v1", "v2"):
    r = run_agent(text, v)
    tc = LLMTestCase(input=text, actual_output=r["reply"],
                     tools_called=[ToolCall(name=c["name"]) for c in r["tool_calls"]],
                     expected_tools=[ToolCall(name="handoff_to_human")])
    m = ToolCorrectnessMetric(include_reason=False)
    m.measure(tc)
    print(v, "score", m.score, "success", m.is_successful())

v1 score 1.0 success True
v2 score 0.0 success False

2. Task completion rate

Precision says whether the calls were right. Task completion says whether the job got done. In this harness a case is complete when every expected call happened and the reply contains the required text. That is a deterministic definition I chose, written down per case. DeepEval's TaskCompletionMetric takes a different route: an LLM judge reads the agent's full trace, which requires setting up tracing. Use the deterministic version wherever "done" can be written as events, and a judge where it cannot.

3. Faithfulness

Faithfulness asks whether the agent's claims are supported by the context it was given. DeepEval's FaithfulnessMetric is an LLM-as-judge metric for retrieval pipelines: it needs the agent's output and the retrieved context, and it checks the output against that context. I did not run it for this page, because it needs a judge model and a key. For a tool-using agent with no retrieval, the cheaper check is deterministic. If the reply says "Booked", a successful book_meeting call must exist. The reply_contains field plus the expected calls above does exactly that.

4. Latency: p50 and p95

Latency matters because a sudden rise often means extra tool calls or a loop. Track the median and the 95th percentile end to end, from the user's message to the final reply. A common snippet computes p95 as sorted_values[int(n * 0.95)]. For 20 values that returns the maximum, not the 95th percentile. Nearest-rank is ceil(0.95 * n) - 1, which gives the 19th value, and a unit test pins it down:

from evals import percentile
def test_p95_nearest_rank():
    assert percentile(list(range(1, 21)), 0.95) == 19   # 20 values: 95th percentile is the 19th, not the max
    assert percentile([5], 0.95) == 5
def test_old_formula_returns_max():
    s = list(range(1, 21)); assert s[min(int(len(s) * 0.95), len(s) - 1)] == 20

2 passed in 0.01s

The stub answers in microseconds, so the latency figures in the run below are meaningless for judging an agent. Measure latency on the real agent, with real tool calls, and set the budget from how long your user will wait.

5. Regression rate

Regression rate is the share of cases that passed on the previous version and fail on this one. It is the number to watch while tuning prompts, because it shows what a change broke even when the overall score looks flat. To test it, I made a second version of the stub (v2) that drops the complaint-handoff rule, a stand-in for a prompt edit that removes a branch by accident. Here is the report script and its output:

import json
from evals import *

cases = load_golden()
v1, v2 = run_suite(cases, "v1"), run_suite(cases, "v2")
for name, rows in (("v1", v1), ("v2", v2)):
    lat = [r["latency_s"] * 1000 for r in rows]
    print(f"{name}: tool_precision={tool_precision(rows):.2f} completion={completion_rate(rows):.2f} "
          f"p50={percentile(lat, .5):.3f}ms p95={percentile(lat, .95):.3f}ms n={len(rows)}")
rate, broke = regression_rate(v1, v2)
print(f"regression v1->v2: {rate:.2f} ({len(broke)} of {sum(r['complete'] for r in v1)} previously passing) {broke}")
with open("failures-v2.jsonl", "w") as f:
    for r in v2:
        if not r["complete"]:
            f.write(json.dumps({"id": r["id"], "input": r["input"], "actual": r["actual"]}) + "\n")

v1: tool_precision=1.00 completion=1.00 p50=0.010ms p95=0.018ms n=20
v2: tool_precision=0.80 completion=0.80 p50=0.008ms p95=0.012ms n=20
regression v1->v2: 0.20 (4 of 20 previously passing) ['hd-01', 'hd-02', 'hd-03', 'hd-04']

The v2 run loses four cases that v1 passed: hd-01 to hd-04. Regression rate is 0.20, tool precision falls from 1.00 to 0.80, and task completion falls from 1.00 to 0.80. The same cases show up under pytest, with one result per case id:

$ pytest tests/ -q
....................                                                     [100%]
20 passed in 0.02s

$ AGENT_VERSION=v2 pytest tests/ -q --tb=no
................FFFF                                                     [100%]
=========================== short test summary info ============================
FAILED tests/test_agent.py::test_case[hd-01] - AssertionError: got [{'name': ...
FAILED tests/test_agent.py::test_case[hd-02] - AssertionError: got [{'name': ...
FAILED tests/test_agent.py::test_case[hd-03] - AssertionError: got [{'name': ...
FAILED tests/test_agent.py::test_case[hd-04] - AssertionError: got [{'name': ...
4 failed, 16 passed in 0.02s

Step 3: read the failure traces, not only the rate

The report also writes each failing case to a JSONL file, so you can read what the agent did instead of guessing. These are the first two lines for v2:

{"id": "hd-01", "input": "This is unacceptable, I want a refund", "actual": [{"name": "qualify_lead", "arguments": {"score": 20, "follow_up": false}}]}
{"id": "hd-02", "input": "I'm angry, nobody replied to my last email", "actual": [{"name": "qualify_lead", "arguments": {"score": 20, "follow_up": false}}]}

The agent called qualify_lead on an angry refund request and never handed off. A pass rate of 0.80 would not tell you that. Look at the split by group for the v2 run: 12 of 12 lead-qualification cases passed, 4 of 4 booking cases passed, and 0 of 4 handoff cases passed. One whole behaviour is gone.

That points at a weakness in gating on the overall rate alone. If only two of the four handoff cases had failed, v2 would score 18 of 20, which is 0.90, and a 90% gate would let it ship with the handoff path half broken. Mark the cases you cannot afford to fail, such as handoffs, refunds and anything that writes to a system, as critical, and block the release on any critical failure whatever the overall rate says.

Step 4: run it in CI

Run the suite on every pull request that touches the agent, its prompts or the golden set. One trap: pytest exits non-zero on any failing case, so a workflow that says "fail below 90%" but runs a single pytest step fails on the first bad case instead. To enforce a rate, let the pytest step produce a JUnit file without failing the job, then check the rate in a second step. Here is the gate script and its result on the two runs above:

import sys, xml.etree.ElementTree as ET
root = ET.parse(sys.argv[1]).getroot()
suite = root if root.tag == "testsuite" else root.find("testsuite")
total, bad = int(suite.get("tests")), int(suite.get("failures")) + int(suite.get("errors"))
rate = (total - bad) / total
print(f"pass rate {rate:.2f} ({total - bad}/{total}), threshold {float(sys.argv[2]):.2f}")
sys.exit(0 if rate >= float(sys.argv[2]) else 1)

$ python gate.py v1.xml 0.90
pass rate 1.00 (20/20), threshold 0.90
exit=0
$ python gate.py v2.xml 0.90
pass rate 0.80 (16/20), threshold 0.90
exit=1

And the workflow that wires it together:

name: Agent Evals
on:
  pull_request:
    paths: ["agent/**", "prompts/**", "tests/**", "golden_set.json"]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install pytest
      - run: python validate_golden.py
      - run: pytest tests/ -q --junitxml=eval-results.xml
        continue-on-error: true
      - run: python gate.py eval-results.xml 0.90
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-results
          path: eval-results.xml

I parsed this file as YAML, but I did not run it on GitHub, so treat it as a starting point. The action versions are the current majors as of 30 October 2026: actions/checkout v7.0.1, actions/setup-python v7.1.0 and actions/upload-artifact v7.0.2, each read from the repository's releases. If you use DeepEval, its docs recommend deepeval test run over plain pytest because it adds flags for repeats, error handling and async. Langfuse documents a GitHub Action, langfuse/experiment-action, that fails the job when your experiment script raises RegressionError.

The 90% threshold here is a starting point I picked, not a standard. A stub is deterministic, so one run is enough. A real LLM agent varies between runs, so run each case more than once before you decide how much slack the rate needs.

Tools compared

These rows were checked on 30 October 2026 against PyPI, the projects' own repositories and Langfuse's pricing page.

The LLM-as-judge problem

A judge model is another model with its own bias. The paper LLM Evaluators Recognize and Favor Their Own Generations (arXiv:2404.13076, April 2024) reports that an LLM evaluator can score its own outputs higher than others' while human annotators rate them as equal in quality. Two practical consequences follow. Use a judge from a different model family than the agent. And before you trust judge scores, label 20 to 30 cases by hand, run the judge on the same cases, and look at how often it agrees with you. I did not run a judge for this page, so I have no agreement figure to show. A judge also adds a model call per case per metric, so judged metrics are better kept out of the per-pull-request gate and run on a schedule instead.

A release checklist

These are starting rules, written as gates. Adjust each to the cost of a failure in your own workflow.

If you are not sure the use case is ready for production at all, the AI readiness assessment covers the organisational side, and the pilot to production guide covers what usually breaks between a demo and a launch. To judge whether the use case is worth the cost of building and running evals, the ROI calculator gives you a first estimate.

What this page does not show

The agent is a stub, so there is no model variance, no real latency and no real tool failures. The golden set is synthetic and hand-written by the same person who wrote the stub. Faithfulness was not run. The workflow file was parsed but not executed on GitHub. Agents that run for hours add another failure class, resuming after a crash, which I cover in durable execution for long-running agent campaigns. Everything above that I did run is shown with its output, and the code is complete enough to copy and adapt to a real agent by replacing run_agent().

If you want a second pair of eyes on which cases and gates make sense for your agent, you can book a call.

Want help setting up evals for your agent?

If you have an agent in production or close to shipping and are not sure where to start with evals, I can review your setup and point out the biggest gaps. Book a free 30-minute call.

Book a Free Discovery Call

Suyash Raj Founder, Voxdonna AI and AiSewak. Writes here about n8n, AI agents, and voice automation.