Agent Evals Must Raise the Floor, Not Chase the Ceiling
Ben Hylak of Raindrop argues that agent teams should focus less on headline capabilities than on raising the “floor”: the product-specific failures that break trust when agents can act through tools, permissions and external systems. Because agents can produce effectively unlimited issues, he says teams need to know when a failure began and what share of users it affects, then evaluate the full harness in code rather than rely on brittle chatbot-era test suites.
The floor, not the ceiling, is where trust breaks
Ben Hylak frames production-agent evaluation around a practical question: not what issues an agent has, but how a team can make it better with the tools available now.
His governing distinction is between an agent’s ceiling and its floor. The ceiling is the most impressive or unexpected capability an agent can display: the creative, emergent behavior that makes agents useful. The floor is the worst thing it does. That might mean recommending a competitor, deleting data, or sending an unwanted low-quality email to a customer because the system happened to have access to email.
Agents’ ability to navigate an environment, use tools, and improvise around roadblocks creates both sides of that distinction. A creative workaround can complete a task a developer did not anticipate. It can also produce a harmful outcome the developer did not imagine. Hylak argues that the floor is what breaks user trust: not merely an imperfect answer, but a failure with consequences in the product’s actual environment.
The distinction also separates the work of product companies from that of frontier-model labs. Labs are pursuing broad capability gains. Product companies have to encode company-specific knowledge: what their data means, which tools an agent can use, what permissions it has, and what happens when it makes a mistake. Companies that borrow the language and shapes of lab benchmarks without making that distinction can end up pursuing a goal that does not fit their responsibilities.
Hylak calls the unexamined choice “benchmark-maxer or floor-raiser.” The right emphasis depends partly on what users can catch and correct themselves. An engineer can delete a bad autocomplete suggestion. More autonomous coding workflows leave more for the user to configure and verify. An AI doctor, he argues, presents a different distribution of responsibility because users may not recognize a wrong answer.
Irreversibility matters as well. An agent that drafts text for a knowledgeable user to review is not the same kind of system as one that sends emails, moves money, deletes data, or makes decisions that persist. Raising the floor means making the product-specific failures that matter most observable, measurable, and actionable.
The practical task is to make unknown failures visible
The question is not whether an agent has issues. Hylak’s premise is that agents can have effectively unlimited problems: they are stochastic systems operating across tools, permissions, external systems, and varying user contexts. Some apparent problems may not be worth solving locally, particularly if a future model improvement can address them more effectively. The immediate task is to determine which failures are new, consequential, and tractable enough to address.
For each issue, Hylak says a team needs two facts: when it started and what share of users it affects.
A start date turns a vague observation into a change question. If an issue began yesterday, or three or four days ago, the team can ask what changed: the model, a deployment, a tool, or another downstream dependency. A long-running issue may still matter, but a recent onset gives an investigation a more immediate direction.
The percentage of users affected establishes a different kind of priority. A behavior that reached three users is not the same operational problem as one that reached 100,000, even if the individual traces look similar. Hylak is not treating low-volume failures as unimportant. An internal enterprise application with five users may provide highly consequential information and need to be exceptionally reliable. His point is that the operating approach should match both the product’s scale and its stakes.
| Run volume | Suggested operating mode |
|---|---|
| 1–100 runs/day | Read raw logs for confusion, frustration, clear misses, repeated prompts, and awkward moments. |
| 100–1,000 runs/day | Turn recurring stumbles into problems the team can discuss, reproduce, and decide whether to fix. |
| 1,000–100,000+ runs/day | Track longer-horizon behaviors such as context loss, ignored tool errors, refusals, and user frustration. |
| High-volume production | Ship fixes behind flags and compare affected issues and signals against production traffic. |
At sufficiently high volume, experimentation becomes practical. A product with a free tier can expose a small sample to a change and compare the resulting signals with production traffic. For a product with five or 10 users, Hylak would not recommend experiments or A/B tests. The relevant discipline is not to copy a high-scale operating model, but to use methods the available traffic can actually support.
The detection layer should be deterministic, in Hylak’s account. Count keyword frequency, tool names, error codes, or refusal phrases; compare them with the previous hour, day, or deployment; then flag the surprising slices. The aim is to create a signal that is fast, cheap, repeatable, and easy to challenge.
A spike in a keyword such as “refund” does not establish that there is a defect. It does, however, create a bounded slice of behavior that can be inspected: relevant traces, the deployment window, raw tool outputs, and the users affected. Hylak’s division of labor is explicit: agents are poor at finding anomalies in an unbounded corpus, but useful at investigating anomalies that have already been made concrete. Given the suspicious slice, an agent can help ask what changed, why it matters, which users were hit, and what classifier to add next.
The harness, not the prompt, is the thing being evaluated
The chatbot-era model of evaluation assumed a comparatively bounded interaction: ask a question, know the expected answer, and check whether the output contains it. Hylak says that fit a period when systems were mostly chatbots and teams could anticipate perhaps 80% or 90% of the questions users would ask.
Agents make that approach fragile. Their behavior is distributed across application code, retrieval, tools, permissions, external systems, product state, and the sequence of actions the agent takes. As Hylak puts it, “the prompt is actually like the whole thing now”: it includes the code, the harness, and everything connected to the agent.
A test that expects one prompt to produce one particular tool call can lose much of its relevance after a model change or a switch to a different harness. Hylak gives the example of a team that builds a thousand-example suite, switches harnesses—such as moving to Claude Code CLI—and finds that 80% of its evals no longer mean much.
His warning is not against evaluation itself. It is against investing months in predefined, brittle test cases that must be extensively rewritten whenever the surrounding system changes. Teams want more safety, he says, but not “theater”: an eval process whose maintenance does not remain useful as models and agent systems evolve.
His alternative is to make offline evals resemble ordinary software tests. Run the real agent path locally, then assert on the behavior that matters: output state, tool calls, files changed, structured data, or final state. The refund-agent example shown in the source checks that a cancellation-and-refund request leads the system to verify eligibility and create a refund. It also checks the expected tool-call sequence. The test evaluates behavior through the harness, rather than merely whether the final prose sounds plausible.
Hylak points to Sentry’s “harness-backed” terminology and OpenAI’s “macro evals,” but treats the label as secondary. The underlying practice is to keep evals as code, near the product they protect, and to test the complete agent loop. In his view, prompt-playground workflows no longer fit the shape of systems whose behavior is entangled with code, tools, retrieval, permissions, and product state.
Clusters describe traces; they do not define the problem
Trace clustering can be useful as exploratory analysis. Embedding traces and examining groups can make a large mass of logs more intelligible. A large cluster may suggest a common pattern; a small one may point to a rarer failure mode.
The problem begins when clusters become a team’s durable issue taxonomy.
Clusters are hard to track reliably over time because their boundaries can change when the clustering process is rerun. The source illustrates the problem as the same data producing different clusters on different days. If the object being counted keeps changing, a trend line cannot reliably show whether one issue is growing or shrinking.
The deeper problem is that the clustering algorithm, rather than the product team, decides what belongs together. A cluster labeled “price issues” might contain a wrong quoted price, a wrong refund calculation, and a wrong product suggestion. Those traces may appear similar at a statistical level, but they can arise from distinct pricing, billing, and recommendation defects. They may need different owners, fixes, and severity assessments.
Hylak’s formulation is that clustering is a description of data, not a definition of issues. A useful issue boundary has to be product-specific. It depends on what a company considers equivalent, how users experience the failure, what tools and data are implicated, and who can act on the result.
His analogy is conventional telemetry: teams do not periodically cluster all ordinary error logs and use the output as their stable error taxonomy. They need categories that can be assigned, measured, and fixed over time.
Explicit classifiers give teams control of issue boundaries
The alternative is to write classifiers for patterns a team has decided matter in its product. Hylak describes this as applying code mode to traces: write classifiers, run them in a sandbox, and execute them at production scale.
The logic can be straightforward. A classifier might identify a loop after repeated tool calls, a missed extraction, or a tool-failure pattern after more than five failures. The important property is not sophistication for its own sake. It is that the team owns the category and can preserve its meaning as the product changes.
That makes the classifier useful for the two measurements Hylak considers essential. A defined issue can be tracked from its onset and measured by the share of users it affects. A shifting cluster cannot provide the same continuity without solving the difficult problem of maintaining stable cluster boundaries over time.
The result is a production workflow built from defined signals rather than a search for a complete catalog of failure modes. Detect a deterministic anomaly. Investigate the relevant slice. Decide whether the observed behavior represents an issue worth owning. Add or refine a classifier when it does.