AI agent evals: what to measure before you ship

The agent worked perfectly in my tests. I ran it against a dozen inputs, the outputs looked right, and I shipped it. Three days later it started calling the wrong tool on inputs that seemed ordinary, hallucinating field names that were not in its context, and completing tasks in the wrong order when the user phrased things slightly differently than I expected. I had no numbers, no baseline, and no way to tell whether a prompt change made things better or worse.

I had been doing vibe checks and calling them testing. That is the mistake nearly every builder makes when they first ship an agent, because agents look different from software. The outputs are text, they vary run to run, and there is no obvious assertion to write. So most people skip evals entirely, or run a handful of manual tests and hope for the best. Then something breaks in production and there is no signal to act on.

I have since added evals to every agent I build before it goes live. This post covers the five metrics I track, how I build and maintain a golden set, and the tools I use to run evals in CI. Everything here is from working on real agents, not from theory. I will show actual code, paste real outputs, and tell you what did not work as well as what did.

If you are working on AI agents and want to ship with confidence rather than hope, this is the workflow I use.

What evals are, and what they are not

Evals are not unit tests in the traditional sense. A unit test checks that a function returns the expected value for a given input. With a deterministic function, that works perfectly. An LLM-based agent is not deterministic: the same prompt can produce slightly different outputs on different runs, and "correct" is often a matter of degree rather than a binary.

Evals are also not human review alone. I do manual review on new agent builds, and it is useful. But I cannot run a hundred manual tests on every prompt change. The feedback loop is too slow. The point of automated evals is to give you a pass rate you can track over time, and to catch regressions before they reach users.

A useful mental model, which I borrowed from Hamel Husain's eval framework, is three levels:

• Level 1: Assertions. Deterministic checks on tool calls, output format, and specific facts. These run fast, cost almost nothing, and catch the most common failures. I run these on every code and prompt change.
• Level 2: Human or LLM review on traces. You log every agent run, then score a sample for quality, tone, and correctness. More expensive, slower, but it catches the subtler failures that assertions miss. I run these on a weekly cadence or after a major prompt rewrite.
• Level 3: A/B or large-scale evaluation. You run two versions of the agent against the same user base and measure real outcomes. This is the most reliable signal but requires production traffic. I only do this once an agent is already earning enough usage to split.

Most builders skip Level 1 entirely and try to go straight to Level 2 or 3. The problem is that Level 1 is the cheapest and catches the most bugs. Get Level 1 right before worrying about anything else.

The five metrics I track

1. Tool call precision

This is the metric I care about most for tool-using agents. Tool call precision measures whether the agent called the right tool with the right arguments. A wrong tool call can have consequences: updating the wrong record, sending a message to the wrong recipient, or triggering an action the user did not ask for.

I measure it as: (number of correct tool calls) / (total tool calls). "Correct" means the agent called the tool the golden set specifies, with the parameter values that match expectations within a defined tolerance.

For example, when I built a lead-qualification agent, I had it call a qualify_lead tool with the lead's score and a boolean for whether to follow up. In my golden set I had 30 inputs with the expected tool name and argument ranges. On the first version, precision was 0.73. After fixing the prompt to be more explicit about the score calculation, it went to 0.93. That number told me the fix was real, not just a feeling.

import json

def check_tool_call(actual: dict, expected: dict) -> bool:
    """Check that actual tool call matches expected name and key args."""
    if actual.get("name") != expected.get("name"):
        return False
    for key, exp_val in expected.get("required_args", {}).items():
        act_val = actual.get("arguments", {}).get(key)
        if isinstance(exp_val, dict) and "range" in exp_val:
            lo, hi = exp_val["range"]
            if act_val is None or not (lo <= act_val <= hi):
                return False
        elif act_val != exp_val:
            return False
    return True

I run this against every test case in the golden set and report the fraction that pass. Below 0.90 on tool precision and I do not ship.

2. Task completion rate

Tool precision tells you whether individual tool calls are correct. Task completion rate tells you whether the agent actually finished what it was supposed to do from end to end. An agent can call the right tool and still fail the task if it stops too early, calls the right tool with the wrong sequencing, or misses a required step.

I define task completion in my golden set by annotating each test case with a set of "must happen" events: specific tools that must be called, specific values that must appear in the final response, or a specific state the world must be in after the agent finishes. If all of those happen, the task is complete.

For a booking agent, the "must happen" set for a confirmation request might be: check_availability called, create_booking called, confirmation message contains booking ID. If any of those is missing, I count it as incomplete.

In my experience, task completion rate drops most when the agent's system prompt is too long and starts truncating context, or when the user's phrasing triggers a different branch than expected. Both are fixable once you can see the number drop.

3. Faithfulness (hallucination rate)

Faithfulness measures whether every factual claim the agent makes is grounded in the context it was given. For a retrieval-augmented agent, that means: did it say anything that was not in the retrieved documents? For a tool-using agent, it means: did it cite or describe results that were not in the tool's response?

This is the hardest metric to check with pure assertions because it requires semantic understanding. I use an LLM-as-judge here: I pass the agent's response and its context to a judge model and ask it to identify claims in the response that have no support in the context. I then compute (unsupported claims) / (total factual claims).

DeepEval has a FaithfulnessMetric that does this automatically against a set of "retrieval context" strings. The score it returns is 0 to 1, where 1 means every claim is supported. I use it directly for RAG agents:

# requires: pip install deepeval
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

metric = FaithfulnessMetric(threshold=0.8, model="gpt-4o-mini")

test_case = LLMTestCase(
    input="What is the refund policy?",
    actual_output=agent_response,
    retrieval_context=retrieved_docs
)

metric.measure(test_case)
print(metric.score, metric.reason)

For tool-using agents without retrieval, I skip the formal faithfulness check and instead assert on specific fields: if the agent says the booking is confirmed, check that create_booking was actually called and returned success. Grounding in tool results is more reliable than grounding in documents.

4. Latency: p50 and p95

Latency is not just a performance metric; it tells you something about agent behavior. An agent that is suddenly 30% slower than baseline might be calling more tools than expected, entering a loop, or processing a longer context window. I track latency not because I need to hit a specific number, but because changes in latency often signal something else is wrong.

I measure wall-clock time from the first token of the user's message to the agent's final response. I track p50 (median) and p95. The p95 tells me about my worst cases, which are often the edge-case inputs in my golden set.

import time

def run_with_timing(agent_fn, inputs: list) -> list:
    results = []
    for inp in inputs:
        start = time.perf_counter()
        output = agent_fn(inp)
        elapsed = time.perf_counter() - start
        results.append({"input": inp, "output": output, "latency_s": elapsed})
    return results

def p95(latencies: list) -> float:
    sorted_l = sorted(latencies)
    idx = int(len(sorted_l) * 0.95)
    return sorted_l[min(idx, len(sorted_l) - 1)]

I set a soft threshold: if my p95 crosses 10 seconds on the golden set, I investigate before shipping. That threshold is specific to my use case. Yours will be different depending on what the agent is doing and who is waiting for it.

5. Regression rate against the golden set

The previous four metrics tell you the absolute quality of your agent on a given version. Regression rate tells you whether a specific change made things worse. It is the metric I run most often because I am constantly tuning prompts.

I define regression rate as: (test cases that passed in the previous run and fail in this run) / (test cases that passed in the previous run). If I change the system prompt and 5 out of 40 previously-passing cases now fail, the regression rate is 0.125 and I need to understand why before shipping.

To track this across runs, I store the pass/fail state of each test case by its ID, keyed by git commit. Then before shipping I compare the current run's results to the last known-good run. This gives me a diff rather than just an absolute score, which is easier to act on.

Building the golden set

The golden set is the hardest part, and skipping it is the most common mistake I see. Without a golden set, evals are just running the agent against random inputs and hoping the output looks reasonable. That is not measurable.

I build the golden set in two stages.

First, I create synthetic cases before shipping. I think through the main scenarios the agent needs to handle: the happy path, the most common edge cases, and any inputs where I already know the agent should behave a specific way. For a 10-tool agent, I usually end up with 25 to 40 synthetic cases at this stage. I write the expected tool calls and expected response properties by hand for each one.

Second, I add real cases from production as they accumulate. When I see an interesting or unexpected interaction in the logs, I add it to the golden set with the correct behavior annotated. Over time, the golden set shifts from synthetic to mostly real, and it becomes more representative of what users actually do.

The format I use is a JSON file per test suite:

[
  {
    "id": "lead-qual-001",
    "input": "I run a 15-person design agency, revenue around $800k, want to automate client onboarding",
    "expected_tool": "qualify_lead",
    "expected_args": {
      "score": {"range": [70, 100]},
      "follow_up": true
    },
    "expected_response_contains": ["onboarding", "automation"]
  },
  {
    "id": "lead-qual-002",
    "input": "just browsing, not sure if I need this",
    "expected_tool": "qualify_lead",
    "expected_args": {
      "score": {"range": [0, 40]},
      "follow_up": false
    }
  }
]

I keep this file in the repo next to the agent code. When the agent's intended behavior changes, I update the golden set to match before making the code change. That way the golden set reflects what I intend, not just what the previous version happened to do.

Running evals with DeepEval and pytest

I use two tools for running evals: DeepEval for the faithfulness and relevancy metrics that require LLM-as-judge, and plain pytest with assertion functions for everything else. The two work well together because DeepEval integrates with pytest natively.

Here is the test file structure I use for a tool-using agent:

import json
import pytest
from pathlib import Path
from my_agent import run_agent  # your agent function

GOLDEN_SET = json.loads(Path("tests/golden_set.json").read_text())

@pytest.mark.parametrize("case", GOLDEN_SET, ids=[c["id"] for c in GOLDEN_SET])
def test_tool_call_precision(case):
    result = run_agent(case["input"])
    tool_calls = result.get("tool_calls", [])
    assert len(tool_calls) > 0, f"No tool calls for: {case['input']}"
    last_call = tool_calls[-1]
    assert last_call["name"] == case["expected_tool"], (
        f"Expected {case['expected_tool']}, got {last_call['name']}"
    )
    for arg, expectation in case.get("expected_args", {}).items():
        actual_val = last_call.get("arguments", {}).get(arg)
        if isinstance(expectation, dict) and "range" in expectation:
            lo, hi = expectation["range"]
            assert lo <= actual_val <= hi, f"{arg}={actual_val} outside [{lo},{hi}]"
        else:
            assert actual_val == expectation, f"{arg}: expected {expectation}, got {actual_val}"

I run this with pytest tests/ -v --tb=short. The parametrize decorator gives me one test result per case, so I can see exactly which cases pass and which fail. The output looks like this:

tests/test_agent.py::test_tool_call_precision[lead-qual-001] PASSED
tests/test_agent.py::test_tool_call_precision[lead-qual-002] PASSED
tests/test_agent.py::test_tool_call_precision[lead-qual-003] FAILED
...
15 passed, 3 failed in 42.3s

The 42-second runtime is because each test case makes a real LLM call. I accept that cost. Running 40 test cases takes about 2 minutes on GPT-4o-mini, which costs roughly $0.08. That is a worthwhile gate before a deployment.

For production monitoring, I also instrument the live agent with Langfuse, which captures every trace including tool calls, latency, token counts, and model responses. When something goes wrong in production, I can pull the trace, reproduce the input in my test suite, and add it to the golden set. The combination of pre-ship evals and production tracing covers both the planned and unplanned failure modes.

Eval frameworks compared

I have tried four different approaches over the past year. Here is what each one is actually good for:

My current setup: pytest for all Level 1 assertion checks, DeepEval for faithfulness and relevancy on the small subset of cases where I need semantic scoring, and Langfuse in production for trace capture. I do not use all three at once in CI; pytest runs on every push, DeepEval runs on a scheduled daily job, and Langfuse is always-on in production.

Putting evals in CI

The eval workflow only pays off if it runs automatically. Running it manually before every deploy is something I will skip when I am in a hurry, and that is exactly when I most need it.

My GitHub Actions job looks like this:

name: Agent Evals
on:
  push:
    branches: [main, staging]
  pull_request:
    paths:
      - "agent/**"
      - "prompts/**"

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/test_agent.py -v --tb=short --junitxml=eval-results.xml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: eval-results.xml

I set the CI job to fail the PR if the test pass rate drops below 90%. Not 100%: LLM outputs vary slightly run to run, and a small number of cases will occasionally fail due to natural variance, not a real regression. Setting the threshold at 90% means a random-variation failure does not block a deploy, but a systematic regression does. I tune this per agent depending on how sensitive the task is.

I also track the pass rate over time. I store each run's results in a simple JSON file committed to the repo, and I have a script that plots pass rate by commit. After a prompt change, I can see immediately whether the rate went up, down, or stayed flat, which is more useful than a single green or red CI result.

The LLM-as-judge calibration problem

LLM-as-judge is useful but it has a failure mode I did not see until I started comparing judge scores against my own manual labels. The judge model tends to agree with the agent model's style, especially if they are from the same family. This means GPT-4o judging GPT-4o outputs will give higher scores than a human would, particularly on fluency and tone.

I calibrate my judge by running it against a set of cases I have already labeled myself. I take 20 to 30 cases from the golden set and manually score each one on the metric I care about, then compare my scores to the judge's scores. If the judge is consistently higher than me by more than 0.15 points, I adjust the threshold I use for pass/fail downward by that offset. This is not perfect, but it keeps the judge from becoming a rubber stamp.

The other issue is that LLM-as-judge is expensive compared to assertions. Each judged case adds one LLM call. For a 50-case golden set evaluated on two metrics, that is 100 extra calls per eval run. I run the judge-based metrics less frequently and reserve assertions for the CI gate.

What I do differently now, and what it changed

Before I added evals, my deploy process was: run the agent manually a few times, look at the outputs, and push if they looked right. After adding evals, my deploy process is: run the 40-case pytest suite, look at the pass rate, review any new failures, and push only if the rate is above 90%.

The difference in practice: I caught a regression in a lead-qualification agent three days before it would have reached production users. I had changed the system prompt to be more concise and the tool call precision dropped from 0.93 to 0.71 on the golden set. Without the eval, I would have shipped that change because the outputs I manually reviewed all looked fine. The regression was only visible on the edge cases I did not think to check by hand.

I have also reduced the time I spend on manual review after prompt changes, because the eval tells me whether the change was structurally good or bad before I read any outputs. That frees up time to look at the interesting cases rather than trying to spot regressions by eye across 50 outputs.

If you want to talk through what evals make sense for your specific agent, book a call. I am happy to look at your setup and tell you where the most useful eval would be.

Want help setting up evals for your agent?

If you have an agent in production or getting close to shipping and are not sure where to start with evals, I can review your setup and point out the biggest gaps. Book a free 30-minute call.

Book a Free Discovery Call

Suyash Raj Founder, Voxdonna AI and AiSewak. Writes here about n8n, AI agents, and voice automation.