AI Agent Evals Turn Confident Errors Into Measurable Failures

Laurie Voss frames the problem as one many teams encounter after shipping an LLM application: it works on the examples developers tried, then fails on an unexpected request. A prompt change may fix one behavior while quietly breaking another. If the only check is to read a few outputs and decide that they “look good,” there is no reliable way to know whether the system improved—or whether it regressed somewhere else.

Ordinary unit tests are a poor fit for many LLM outputs. The same prompt can produce different wording on each run, and two very different answers can both be acceptable. There is no single expected string to compare against. Human review helps, but it does not scale to every output or run in CI. Voss’s alternative is to make quality testable through a combination of recorded traces and evaluations: traces show what the agent did; evals judge whether it did a good job.

That distinction matters especially for agents, which chain together model calls, tool calls, and decisions. A mistake early in the chain can shape every step that follows. An agent might retrieve the wrong information, reason badly over correct information, invent a number, or answer a different question than the user asked. These failures can all produce fluent, well-formatted reports.

Voss’s warning is concise: “Confidently wrong is worse than obviously wrong.”

Her financial analyst example makes the risk concrete. The demo agent researches a stock ticker and a requested focus, then writes a report from what it found. In one run, it combined analyst percentages from two different search results and presented them as a single consensus: 50% Buy, 52.6% Hold, and 13.1% Sell. Those figures add up to 116%. The individual search results were not the problem; the agent’s reasoning about how to combine them was. A check for grammar, tone, or formatting would not catch that.

Another failure involved the agent asking to save its report to a file. The writing prompt said to write a report but did not specify where it should go or that it should be returned in the response. In the trace, Voss saw the agent attempt a Write action and fail. The demo’s configuration included an unattended permission setting, dontAsk; Voss noted that the agent was told not to ask questions and then made an unhelpful attempt to write a file. The trace exposed the attempted action and failure, while the prompt’s ambiguity left the agent unclear about what “write” meant.

These examples illustrate why a failed output is a symptom, not a diagnosis. The cause might be retrieval, reasoning, hallucination, or a mismatch in scope; each calls for a different response. Without inspecting what happened inside the agent, developers may not know what to change.

Voss uses a practical analogy: traces are logs for AI; evals are tests for AI. A trace is made up of spans, each representing a step such as an agent turn, an LLM call, or a tool call. A span can record its input and output, timing, token use, and sometimes cost. A complete agent turn can contain nested spans for the steps that made it up.

The point is not to produce a more elaborate transcript. A trace lets a developer move from a surprising answer to the operations that produced it. In the workshop’s agent, the research turn searches the web and the writing turn resumes the same session to use the research. Phoenix displayed those turns, the individual web searches, their queries and results, and the agent’s attempted file write.

Voss instrumented the TypeScript demo using Phoenix’s instrumentation for the Claude Agent SDK. One implementation detail mattered: the SDK’s ES module exports are read-only, so manuallyInstrument returns an instrumented copy that the application must use. Importing directly from the original package can leave the agent running without producing traces.

The workshop agent is deliberately small and imperfect. It starts with brief research and writing prompts, uses web search, and runs on Claude Haiku 4.5, which Voss chose because it was fast, inexpensive, and predictably prone to useful demo mistakes. One query call is one agent turn; passing the resulting session ID into the next turn lets the writer continue with the researcher’s context. Wrapping both turns in a parent trace makes the overall run easier to inspect.

In one live run, the writer interpreted “write a concise financial report” as a possible instruction to create a file. The trace revealed the attempted Write call and the failure. Without that view, the user might have seen only an unhelpful response asking for permission. The trace made the agent’s action inspectable; Voss also pointed to the ambiguous prompt and the unattended permission setting as relevant to the behavior.

Voss’s advice is to collect more than one run before drawing conclusions. The workshop’s batch contained 15 queries, including familiar companies and deliberate edge cases: a comparison of Apple and Microsoft, Rivian with less data available, Coca-Cola as a different kind of company from a technology firm, an invented ticker, and a colloquial Tesla question—“yo is it a buy rn.” These examples were designed to expose possible weaknesses, but Voss also stressed that a small workshop dataset is only a starting point. Real traffic is the better source of test examples once an application has users.

Before writing evals, she recommends reading a dozen or more traces end to end. Look at the input, the output, and what specifically went wrong. This manual pass helps reveal failures developers did not anticipate. If teams automate first, Voss warns, they may end up measuring whatever is easy to measure rather than what matters.

For larger collections, a coding agent can help inspect traces and propose categories. But that does not remove the need for human sampling: the coding agent may miss the failure the team cares about, or classify it in an unhelpful way. Voss describes a useful sequence: make a plain-language note about each trace first, group those notes into categories second, and count the categories last. Starting with preselected labels is tempting, but can force unfamiliar failures into categories that are too broad, too narrow, or simply wrong.

In the demo, one category was “harness leakage”: repeated attempts by the agent to request permissions or write files in a setup that did not allow it. Other traces showed hallucinations and reasoning errors, including the mismatched analyst percentages. Frequency helps identify common problems, but severity also matters. Voss’s shorthand is frequency multiplied by severity: a rare but dangerous failure can outrank a frequent minor one.

Voss divides evaluations into code checks, LLM judges, and human review. They are complementary layers, not alternatives.

Code evals are deterministic functions. They are fast, cost nothing to run, and return the same result for the same input. They are suitable for questions with a clear mechanical answer: Did the output parse as JSON? Is it below a length limit? Does it contain a required field or avoid a banned phrase? A code eval can also call a database or API to check whether a state or value is correct; it is not limited to searching for strings.

The workshop’s first check asks whether a financial report mentions the requested ticker. If a report requested for Tesla is actually about Microsoft, that is plainly a failure, and an LLM is unnecessary to detect it. The eval extracts the ticker from the structured input and checks whether it appears in the output. When the workshop ran the check across 17 reports, it found 15 passes and two failures: TSLA and AMZN were missing. Voss said the Amazon report had focused on AWS profitability rather than the broader company report the query called for.

The structured input makes this check easy to write. If the ticker and the report focus arrive as separate fields, the evaluator can inspect them directly. If inputs are unstructured, a system may need another model call to infer what the user asked.

Voss’s guidance is generally to grade the outcome rather than require the agent to follow a particular path. The ticker check does not care how the agent found its information; it checks whether the report is about the requested company. Voss also noted that there are cases where the trajectory itself matters, but that is a distinct kind of evaluation.

When the question depends on meaning—whether the report is accurate, grounded in its sources, relevant, or useful—a second LLM can act as a judge. It grades the output against explicit criteria and can return a score, a label, and an explanation. The explanation is particularly useful: it can identify what was wrong or missing, and help the team recognize a pattern across many failures.

But a judge is another model, with model-like limitations. It can be inconsistent, make mistakes, and be confidently wrong. It also costs money to run. Voss recommends using a judge that is more capable than the model being evaluated, and says teams can reduce same-model preference by using a judge from a different provider. Human domain experts remain important for unexpected failure types and for checking whether the judge’s decisions make sense.

Phoenix’s built-in evals provide a starting point for common dimensions, including correctness, faithfulness, completeness, hallucination, refusal, toxicity, PII, and tool use. A tool-selection eval can ask whether the agent chose a suitable tool; a tool-invocation eval can examine the arguments it passed; and a tool-response-handling eval can examine what it did with the result. Voss described these as prompts and evaluators that had been written and tested for common cases, not as guarantees of correctness.

The most important choice is whether the evaluator is asking a question it can answer from the context it has. The built-in correctness evaluator failed all 16 of the demo’s reports. Its judge did not have the current financial data needed to establish whether the reports were correct. Seven explanations explicitly said the judge could not verify fresh data; other explanations raised real problems, such as numbers that did not add up. The all-zero result therefore mixed a limitation of the judge with actual concerns and did not provide a useful overall assessment.

Faithfulness asked a more answerable question: given the research and search results available to the agent, did the report stick to them? That context was already present in the trace. An early version of the evaluator included research notes but not the search results; the judge then flagged numbers as unsupported even when the agent had found them. Adding the raw search results gave it the evidence needed to make the comparison.

The resulting faithfulness eval marked 14 of 16 reports as unfaithful. More importantly, its explanations pointed to specific problems: a Microsoft report included claims about Azure, backlog visibility, and implied targets that the context did not support; an Nvidia report conflated two data points. The score mattered, but the explanation made the results useful for deciding what to investigate.

Some requirements are specific to the application. The workshop’s financial report should not merely summarize information; it should offer an actionable investment recommendation. There was no built-in eval for that requirement, so Voss wrote a custom judge.

She outlines five parts for an evaluation prompt:

1. Give the judge a role and context. Explain what it is evaluating and what decision it needs to make. In this case, it is judging a financial report for actionable investment guidance, not just a data summary.
2. State explicit criteria. “Helpful and accurate” is too vague to reliably answer a yes-or-no question. The workshop’s rubric said an actionable report makes a specific recommendation, names concrete risks with supporting data, looks forward, and explains why it recommends what it does. A report that only summarizes the past or gives no next step would not qualify.
3. Mark the data clearly. The prompt uses separate XML-style tags for the request and the report. This distinguishes the evaluated material from the judge’s instructions and, when examples are included, helps distinguish examples from the data under review.
4. Consider labeled examples. Examples can make the intended boundary clearer, but can also make a judge overfit to their details. Voss tested the judge both with and without examples instead of assuming they helped.
5. Constrain the answer. The judge should choose between defined outcomes—in this case, “actionable” or “not actionable”—rather than return an unrestricted response.

The examples did not improve this judge’s agreement with Voss. With examples, it agreed with her on 15 of 16 reports in each of three runs; without them, it agreed on 16 of 16 twice and 15 of 16 once. The disagreement exposed a substantive ambiguity. One report recommended Microsoft but contained no risk section. Voss had labeled it actionable; her rubric said concrete risks should be named. The judge treated that criterion more strictly and rejected the report. Voss was not sure that her human judgment was better.

A separate prompt edit demonstrated how sensitive evals can be to wording and structure. Voss had initially left examples as plain text rather than marking them clearly. With the examples untagged, the judge repeatedly returned “not actionable”; wrapping them in XML tags flipped the results to “actionable” on the same data and instructions. The judge had been confused about what was an example and what it was evaluating. Her conclusion: treat eval prompts like code—version them, review them, and test them.

She also argues against a single “God evaluator” that attempts to grade accuracy, tone, completeness, formatting, and policy in one prompt. If it fails, it can be hard to tell which dimension caused the failure, and adding many possible labels may only make the result harder to interpret. Separate evaluators make it easier to see what needs attention: one for faithfulness, one for actionability, one for correctness, for example.

Finally, teams should distinguish release guardrails from longer-term targets. A report inventing a stock price might be grounds to block a release. A report that could suggest other stocks worth considering might be a desirable improvement without being a release requirement. The threshold should reflect what the team considers unacceptable, not merely what is easy to count.

A custom evaluator is a classifier: it sees a report and predicts a label. To assess it, compare those predictions with human judgments made against the same criteria. Voss labeled the workshop’s 16 reports herself and attached the labels to their traces in Phoenix. In a production setting, she says, this is work for a domain expert.

Those labels form a golden dataset: examples where the team has recorded what it believes the right answer is. They are useful only if the human labels are consistent with the rubric. Voss found that she had sometimes been looser than her own rule about concrete risks. On one report, she accepted an answer without risks even though the criterion required them. She concluded that the judge’s stricter decision might have been more faithful to the written rules than her initial label.

Labeling also takes attention. Voss warns that reviewers become fatigued as they read more traces and may start clicking through without applying the rubric carefully. A smaller set of well-labeled examples can be more valuable than a larger, careless one. She recommends continuously adding tricky cases and production failures, while making sure the tasks are ones a human could pass and that each has a known-good answer demonstrating that the task is achievable.

The judge and human labels can then be compared on a development split, with a separate test split held back. In the workshop, the 16 reports were sorted by span ID before every fourth example was assigned to the test split. That left 12 reports in development and four in the held-out test split. The judge agreed with Voss on all 12 development reports. That was encouraging, but not strong evidence: the development set was small, and the test set contained only four reports.

One of the useful outcomes was that uncertainty in the human label was visible rather than hidden. Voss had hesitated over a Coca-Cola report that called the stock “ideal for conservative income-oriented portfolios.” She ultimately decided that this was not a sufficiently explicit recommendation, and the judge also treated the case as borderline. A Microsoft report presented the reverse tension: it said “buy” and offered reasons, but named no risks. The judge’s explanation said the report cleared the actionability bar, but only narrowly, despite the weak risk treatment and absence of a risk section. Voss’s rubric required concrete risks, and she had initially accepted the report. These are not merely disagreements to average away. They can reveal that the criteria need clarification—or that the team itself has not settled what quality means.

A proposed clarification made the evaluator worse on the available examples. Voss added that a report was actionable only if it met all four criteria. This was a change to the judge’s rubric, not a new set of human labels. With the change, judge–human agreement on the 12-report development split fell from 12/12 to 11/12; on the four-report held-out test split, it fell from 4/4 to 3/4. The stricter wording caused the judge to reject a Microsoft report in development and an Apple report in test. Making a rubric more explicit is not automatically an improvement; it can make the evaluator less aligned with the intended human judgment.

Voss discusses precision and recall as different ways to understand a judge’s mistakes. Precision asks: when the judge says a report fails, how often is that right? Recall asks: of the reports that really fail, how many did the judge catch? In many settings, she says, recall deserves priority. A false alarm costs time spent reviewing an acceptable report; a missed failure can reach a customer. The right balance depends on the consequences of each type of mistake.

Judges have biases that also deserve attention. Voss says they may favor the first or last option in a comparison, prefer longer answers, be persuaded by confident language, or favor outputs from their own model family. Human agreement is not a perfect ceiling either. Voss cited Cohen’s kappa values of 0.2 to 0.3 for this kind of task, while cautioning that human reviewers also disagree. The practical aim is not necessarily perfect agreement with one reviewer, but performance that is useful against a carefully applied human standard.

A failure should also seem fair. If a report marked as bad looks acceptable under the stated criteria, the evaluator may be the problem. Voss described a benchmark example in which an answer with four digits of precision was rejected where the test expected six, despite the value being right to the precision supplied. She also cited ambiguous task descriptions and tasks that could not be reproduced. After the benchmark’s tests were reviewed and corrected, she said its score rose from 42% to 95%. The lesson for application evals is to keep examining the test set, not assume that every failure belongs to the agent.

Once teams have an evaluator they can use, they can test changes with a controlled comparison: the same inputs, the same evals, before and after. A prompt change tested on a couple of outputs by eye is still an intuition, not evidence about its broader effects.

In the workshop, Voss assembled reports into a dataset for experiments. She recommends keeping both failures and passes. A failure set helps test whether a change addresses a known problem; passing examples help reveal whether the fix broke behavior that already worked. Phoenix experiments run a task on dataset examples, score the results with evaluators, and store them for comparison. The task can run the whole agent or a narrower behavior.

The revised writing prompt mapped each change to an observed failure. It required an explicit buy, hold, or sell recommendation with reasons; concrete risks supported by data; and a forward-looking view. It also added a comparison section for multiple tickers, told the agent to use only numbers found in its research, and asked it to return the report as its reply rather than write it to a file.

The revised prompt was run three times against the 16-report set, producing 48 results. The original figures below are pass counts from one run across 16 reports; the revised figures are pass counts across three runs:

The changed prompt improved the measured actionability and ticker results, and the five reports that had previously passed continued to pass. Faithfulness barely improved. Despite the instruction to use only numbers in the research, the agent continued to invent or misuse information. Voss’s interpretation was specific: the prompt fix solved the behavior the team measured as actionability, but did not solve the separate faithfulness problem. That became the next issue to investigate.

Because agent outputs vary, a single run per example can mislead. Voss distinguishes pass@k—succeeding at least once across k tries—from pass^k—succeeding every time. A coding assistant that gets a task right on its second attempt may be acceptable; a financial analyst that gives inaccurate recommendations one time in five may not be. For a high-stakes use, consistent success across repeated runs may matter more than the ability to succeed once.

The workshop also added a consistency check for recommendations across repeated runs. Voss reported that eight of 16 questions got the same call every time. The point was to expose a kind of variability a single output would not reveal: the same question could result in different advice. The workshop’s small set of examples makes this a diagnostic result, not a firm estimate of how often the agent would vary in production.

Voss’s suggested order for improving an agent is to address data quality first, then prompts, then model choice, and finally hyperparameters. A poor or hard-to-interpret input can undermine both the agent and the evaluator. Prompt changes come next because they target the behavior directly. Model choice can have considerable effect but also changes many things at once, often with a cost trade-off. Hyperparameters such as temperature are, in her view, usually the least useful place to start.

Voss compared Haiku 4.5 with Haiku 5.5 using the evaluation results from the experiment. The newer model improved faithfulness and recommendation consistency, while leaving the ticker and actionability checks unchanged. Its cost per run also rose substantially.

Faithfulness rose from three to ten passes out of 48, but remained poor. Consistency rose from eight to 14 questions out of 16, while cost per run increased from $0.30 to $1.17.

The consistency improvement had a complication: Haiku 5.5 often gave the same recommendation because it tended to choose “hold.” Across 45 runs on real tickers, Voss reported Hold 41 times, Buy four times, and Sell zero times. Repeating the same conservative answer can look good to a consistency metric without necessarily producing better advice. Voss thought the model had become too conservative.

The experiment did not settle whether Haiku 5.5 was the right model. It made the trade-off legible: improved faithfulness and repeatability, a large cost increase, and a tendency to give the same cautious answer. That is the value of measuring before switching. The numbers do not make the decision, but they show what changed and give the team a basis for deciding what to investigate next.

A capability eval describes something the agent cannot yet do reliably and gives the team a target to work toward. When the agent can pass that task consistently, the eval can become a regression check: a behavior that used to work should continue to work. Voss also describes eval-driven development, where a team writes the eval before building a feature so that “done” has a measurable definition from the outset.

The workshop uses Vitest, a JavaScript test runner, to run the agent and its existing evals against a regression set. Each test runs a query, records the report, and evaluates it for ticker presence and actionability. The checks do not need to be rewritten for CI; they are the same evaluators used during development.

Not every result needs to be a hard release gate. The demo treats mentioning the requested ticker as a guardrail: every report must be about the right company. Actionability is a target, with a minimum average score of 0.8. Because an LLM judge can be noisy on an individual report, the build gates on the average rather than failing on one low-scoring output.

A red run can therefore report more than a pass or fail. In Voss’s example, all four reports mentioned their requested ticker, but the Rivian report was not actionable. The actionability average was 0.75, below the 0.8 threshold. The test output identified the failing report and the reason the threshold was missed. Voss’s point is that the explanation is part of the operational value: a red cross tells the team something failed; an attached evaluation gives it a place to start.

The same suite can run in GitHub Actions with Phoenix as a service container. But Voss cautions against running the entire agent on every commit if it makes the test cycle too slow or expensive. Teams can use smaller, targeted checks during development and reserve broader experiments for the questions that justify them.

Once the application is live, the loop continues. Run evals on a sample of real traffic, watch for scores that drop and stay down, and alert on trends rather than every isolated failure. A production failure becomes an example in the dataset; the team can inspect the trace, refine its evals or system, and test the next change against that accumulated experience.

The workshop’s loop is to trace the agent, read what happened, evaluate it, check the judge against human judgment, improve the system through experiments, and lock in reliable behavior with CI. Each production failure can become another example and send the process around again.

The starting point need not be the full workshop. Voss’s proposed first steps are to read about 20 traces, write one deterministic code eval, and add one LLM eval for the most important quality dimension. Then run them, inspect the results, and build from there. The larger discipline is to keep returning to the evidence: trace what the agent did, read what happened, measure what matters, check whether the judge is credible, test changes against the same examples, and preserve the behaviors that are already working.