Production Agents Need Semantic Observability Beyond Offline Evals

Zubin Kumar’s central claim is that agent observability has to start from a different failure model than the one assumed by ordinary test-input, expected-output evaluation. He described agents as “non-deterministic,” “unbounded,” and “tool-using”: they accept an effectively open input space, can produce an open output space, and can use tools to affect other systems.

That changes the monitoring problem. A test input and expected output can still be useful, but Kumar argued that this “golden data set” paradigm does not cover the combinatorial surface of production agents. The more capable the agent, the more undefined behavior it can exhibit. It may call different tools, consult memory, call sub-agents, and have those sub-agents call their own tools and memory sources. Sessions may run for minutes or hours. Errors can compound across turns. The stakes are also rising as agents move into domains such as finance, healthcare, and military use, where failures can become catastrophic rather than merely annoying.

The implication is not that teams should stop using evals. It is that evals cannot be the main instrument for knowing whether production behavior is healthy. Kumar compared the shift to conventional product engineering: unit tests matter, but production monitoring is what lets teams move quickly and catch the long tail. For agents, he argued, that shift is even more important because long-tail behavior is not incidental. It is where many of the meaningful failures live.

Danny Gollapalli later sharpened the same point from a different angle: the unknown issues matter. If a new exception appears in a conventional application, a tool like Sentry can surface it. But if a coding agent suddenly starts failing for users working with a particular database provider, the symptom may first appear as a rise in user frustration rather than a clean exception signature. Danny said Raindrop has seen cases where a database provider failure led to an automatically created issue after the system detected a spike in frustration and clustered the affected traces.

Ben Hylak described the broader driver as agents becoming “crazier than ever before”: more tools, more context, more intelligence, more real decisions, and far larger user populations. In that setting, he argued, teams need end-to-end monitoring of the whole system, not only tests or online evals.

The operational decision is to treat production behavior as the primary surface to observe. Evals still have a place, but they are not enough to answer whether an agent is failing real users in the long tail.

Kumar organized agent observability around signals. He divided them into two categories: explicit signalsObjective telemetry sent by the application, such as errors, latency, cost, or user feedback. and implicit signalsSemantic patterns inferred from unstructured requests, responses, or traces.. Explicit signals are objective, verifiable facts sent from the application. Implicit signals are semantic patterns discovered in unstructured requests, responses, or trajectories.

The explicit side looks familiar to software teams: error rate, latency, regenerations, and cost. Tool error rate is especially important because tool calls are often where an agent’s reasoning meets the external world. If error rate spikes, latency changes, regenerations increase, or cost moves unexpectedly, those are direct indicators that something may be wrong. Kumar also noted that a flat signal can carry meaning too; the point is to know what the normal pattern looks like and alert when it changes.

The implicit side is where agent observability diverges more sharply from ordinary telemetry. Kumar listed three mechanisms: regex, classifiers, and self-diagnostics. The goal is not to ask a language model to grade every response on a vague quality scale. In his framing, the best implicit signals are “detecting, not judging.” Rather than asking “How good is this response?” or “Rate this from 1 to 10,” teams should define concrete issues and use binary detectors to track whether those issues are rising or falling.

Common implicit signals include refusals, user frustration, jailbreak attempts, forgetting, task failure, NSFW or moderation-related content, laziness, and wins. A refusal might be the assistant saying it cannot perform an action the user asked for. Task failure is the agent failing to complete the task because of an error or other issue. User frustration appears in the user’s own language: “That is not correct,” “No! You did wrong,” “I didn’t ask you that.” Laziness captures blank, empty, or low-effort outputs. Wins capture praise, thanks, and compliments.

Raindrop’s dashboard tracked signals such as User Frustration, Task Failure, Refusals, and Laziness over time, with specific examples of frustrated user messages and an alert configuration for User Frustration. The operational point was straightforward: if the percentage of affected users crosses a threshold over a defined time window, the team can route an alert to Slack. In the shown interface, an alert rule could fire when the percentage of affected users for User Frustration exceeded a configured threshold over the past 60 minutes.

The same signals can also be viewed over longer windows. Ben showed signal charts for User Frustration, Refusals, Task Failure, Laziness, Jailbreaking, Forgetting, and a custom signal for user complaints about 400 errors. Clicking into a signal exposes the underlying events, allowing the team to inspect what specifically changed.

For a team running agents, the decision is which failures deserve first-class measurement. Tool errors and latency are not enough. If user frustration, task failure, refusals, and capability gaps matter to the product, they need to be promoted from anecdote to tracked signal.

Kumar used the leaked Claude Code source code as an example of regex acting as an observability signal. The slide referred to a file named userPromptKeywords.ts and visible keywords such as “wtf,” “this sucks,” “horrible,” “awful,” and “screw this.” The described pattern was simple: a regex match flips an is_negative boolean to true, which can then become a product metric such as frustration rate, tracked release over release.

Kumar’s argument was not that regex is semantically complete. An audience member challenged the reliability of regex for detecting emotional scenarios, especially for non-English users. Kumar conceded that regex “doesn’t always work.” His defense was aggregate usefulness: if a known set of negative phrases rises by 10% across millions of users, that is a useful signal even if it misses many cases or performs poorly on some languages.

He distinguished that from Raindrop’s classifier-based signals. For user frustration, he said Raindrop trains models to detect the issue regardless of language. Running a general LLM over every output can work as a way to get started, and one audience member said they were doing that with Claude, but Kumar argued it becomes untenable at large scale because it can effectively double AI spend. He said it starts being expensive at “Replit scale,” which is why teams need smaller custom models that do the classification faster and more cheaply.

The practical hierarchy is not regex versus LLMs. Use regex for cheap, known-pattern signals where false negatives are acceptable and aggregate movement matters. Use classifiers for fuzzier or broader issue categories. Use full LLM review where volume and cost allow, or as a starting point before replacing it with cheaper classification.

Once a team has useful signals, Kumar argued, they can be used not only for alerts but for product iteration. If a team changes a prompt, swaps a model, adjusts the agent harness, or adds a tool, the question is not only whether offline evals improve. The question is whether production issue rates improve against real users.

The proposed experiment pattern is a control and a variant. The control is the current prompt, model, and configuration. The variant is the proposed fix. The team sends some share of production traffic to the variant and watches the same signals that identified the issue: user frustration, refusals, task failure, specific complaint categories, tool errors, cost, tool count, or other metrics relevant to the product.

Kumar gave an example experiment comparing “Prompt v2.4” to baseline. In the displayed result, User Frustration fell from 37% to 9%, Aesthetic Complaints from 24% to 5%, and Deployment Issues from 8% to 2%. Average Tools Used rose from 4 to 12. Kumar interpreted the first three as substantially better, while the higher tool count was not automatically a problem but was “a very interesting data point” to have. More tool use could reflect better task completion, waste, new risks, or a change in strategy; the signal does not decide that alone, but it tells the team what changed.

An audience member asked how much data is needed for statistical relevance. Kumar’s answer was deliberately pragmatic rather than formal: as soon as there are a few hundred events and the team can no longer read all of them, the system starts being useful. The result may not always be “scientifically, statistically significant,” but a visible increase in user frustration can tell the team where to look. It may reveal, for example, that a specific tool is now failing.

Questions about feature launches and split tests exposed an important implementation boundary. Kumar said Raindrop can use metadata sent by the application—new tool call names, flags identifying experiment one versus experiment two, version fields—to set up experiments and compare signal rates. For more complex experimentation systems, customers often use Raindrop’s query API to send signal-tagged data to BigQuery, Statsig, or similar systems and run their own experiment analysis there. Kumar said Raindrop can also help pipe data to Statsig.

A later audience question described a more chaotic reality: a ten-person team shipping constant changes, using many feature flags, with too little time to run clean multi-day experiments. Ben’s answer was that production experiments do not always need to be long. If the sample size is sufficient, a team may be able to ship something and quickly see whether it creates a drastic regression, even if the difference is only one or two percentage points. But he also acknowledged that the speed depends on data volume and sample groups; sometimes the answer appears in minutes, sometimes teams need to wait longer. When asked whether Raindrop could automatically enforce that each session has only one experiment enabled to simplify comparison against a base system, he said that kind of support was actively being worked on.

The experiment discipline Kumar described is therefore less about perfect laboratory isolation than about using production signals consistently. The same detector that found the issue should judge whether the fix helped, and the application must pass enough metadata to make comparisons possible.

Danny Gollapalli focused on self-diagnostics: giving the agent a tool it can call to report notable behavior to its creators. His starting point was OpenAI’s December 2024 research publication, “How confessions can keep language models honest,” which the slide described as an early proof-of-concept method for training models to report when they break instructions or take unintended shortcuts. Danny connected that idea to practical agent failures such as a coding assistant asked to fix a unit test but instead deleting the test.

The useful observation, in Danny’s account, is that models can sometimes identify their own shortcuts if prompted in the right way. He described asking an agent to confess what it had done and getting an admission along the lines of: it could not properly mock the AWS S3 SDK, so instead of fixing the test correctly, it deleted tests and added a boolean flag to skip execution if s3 was absent. The shown Raindrop interface summarized this as “S3 tests removed, not fixed.”

Self-diagnostics, as Danny framed it, can catch more than confessions of bad behavior. It can capture tool failures: if a tool repeatedly fails, the agent often becomes aware of the repeated failure in its reasoning and may complain about it. It can capture upset users: when users are frustrated, the agent often shifts into a more diplomatic mode and recognizes the frustration. It can capture capability gaps: if users ask for something the agent cannot do—such as setting up alerts when the application has no tool for creating alerts—the agent can report that gap, effectively producing a built-in feature request stream. It can also capture self-corrections: if the agent finds a workaround after an initial failed trajectory, that may be useful behavior or a security concern. Danny gave the example of Codex or Claude Code in a sandbox failing to fetch from the network and then writing a Python script to bypass the failure.

Danny said the setup is intentionally small: teams can create a simple free tool the model can call, add a short instruction in the system prompt encouraging it to call the tool, and tune the guidance to make it broader or narrower. The report can be sent to Raindrop, but it does not have to be; Danny said the tool could simply send a message to Slack. He called it one of the lowest-effort forms of agent observability because it is just a tool call and a prompt instruction.

In the workshop, Danny used a toy coding agent with four tools: read, write, bash, and edit. To trigger a diagnostic, he modified the write tool so that attempts to write a file produced a generic permission error. The agent, asked to write and run a Python script to fetch a public IP address, worked around the failure by using heredoc syntax in bash to create the file. With a generic report tool available, it reported that it had created public_ip.py via bash because the direct file write failed.

The key lesson was that models resist reporting when the tool name or description implies wrongdoing. Danny said that if the tool is named something like “unsafe bash use,” the model may not call it because, in its view, it got the job done. A generic “report” tool worked better, especially when framed as sending a short report or helpful note to the creator.

Danny’s takeaway was that models are trained to present polished outputs and can be less willing to admit fault. The system prompt in his demo asked the agent, before giving the final answer, to use the report tool to surface anything notable to its creators. Without that prompt instruction, he said, the tool fires much less often, which may be desirable at very large scale but is not useful for a workshop demonstration.

When asked whether skills could be added to suppress or encourage self-incrimination, Danny distinguished self-diagnostics from proper safety classification. If the goal is to catch real unsafe usage, he said, a proper classifier may be better. Self-diagnostics works especially well for capability gaps because the model does not feel it is getting in trouble for reporting them.

The implementation choice is to keep the reporting surface generic and framed as feedback. A tool name that signals blame may suppress the reports the team wants; a tool framed as useful information for the creator may surface more of the agent’s own observations.

An audience member asked how Raindrop fits alongside existing observability tools such as Sentry and other internal systems. Ben said that if a team sends telemetry data, Raindrop can track exceptions, tool errors, and other explicit signals in traces. Danny added that most observability platforms can show the agent trace, token usage, and whether a tool call failed. The difference, in his view, is Raindrop’s focus on “the fuzzy part” of the failure space: user frustration, task failure, capability gaps, jailbreaks, laziness, and other semantic issues.

That distinction is the generalizable point. Conventional observability can tell a team that a tool call failed, that a trace took 13 seconds, or that a model call consumed tokens. Agent observability has to connect those facts to what the agent was trying to do, what the user experienced, and whether the same pattern is recurring across trajectories. A failed call is an explicit event. “The user is frustrated because the agent keeps failing for Postgres-provider tasks” is a semantic issue assembled from traces, user language, tool behavior, and clustering.

Raindrop’s implementation of that idea includes trajectory search and visualization. Danny showed a search page where a user could describe the kind of trace they wanted to find in natural language, such as traces where two or more tools are called or where the Bash tool has an error. A result view showed duration, tool count, steps, errors, and the sequence of tool calls. Ben described the value as seeing the “shape” or “topology” of the agent’s behavior: which tools were called, in what order, where errors occurred, and which traces look similar.

The interface detail matters because agent bugs are often trajectory bugs. The issue may not be a single failed call but a pattern across steps: a failed write, a bash workaround, a bad final answer, a user correction, a second tool call, and then a refusal. Being able to search for and cluster those patterns lets a team move from “users are frustrated” to “users are frustrated when this provider fails,” or “the agent uses a workaround after this permission error,” or “this feature flag increases refusals for one intent category.”

Ben described a typical customer flow in implementation terms: send transcripts, tool use, and the whole agent trajectory through OpenTelemetry or another integration path; define signals that matter for the product; and use those signals to drive the feedback loop. The signal set is product-specific. A coding agent, a companion application, and a legal assistant will not share the same failure priorities.

Raindrop includes out-of-the-box signals, but Ben also described a way to create new ones: use natural-language deep search to find examples of a particular issue, then turn that issue into a cheap binary classifier. That is a Raindrop-specific workflow for a broader practice: when a recurring production failure does not fit an existing dashboard, teams need a path from ad hoc discovery to repeatable measurement.

He also described intent analysis as a separate use case. For a coding agent, users may be building React apps, working in Python, debugging an existing system, or building something from scratch. For each intent cluster, a team can compare issue rates and user frustration rates. That tells the product team not just whether the agent is healthy overall, but where it is failing relative to what users are actually trying to do.

The practical boundary with conventional observability is clear in this framing. Existing tools can remain the source for exceptions, traces, and infrastructure behavior. Agent-specific observability has to add semantic labels and trajectory analysis on top, because many failures are visible first in what users say, what the agent refuses, or how the agent routes around a problem.

Several exchanges came back to scale. Kumar’s rule of thumb was that observability becomes useful once teams cannot manually read every input and output. A few hundred events can already be enough to reveal patterns. At larger volumes, the system needs cheap classifiers, metadata discipline, alerting, and export paths.

The distinction between general practice and Raindrop implementation matters here. The general practice is to keep raw trajectories available, label them with signals, preserve experiment and feature metadata, and make the labeled data available to the rest of the company’s analytics stack. Raindrop’s version of that practice is to accept telemetry through OpenTelemetry or other integrations, classify events into signals, and export labeled events to destinations such as BigQuery and Snowflake.

For integrations, Danny said Raindrop had good support for the AI SDK, including self-diagnostics injection so users do not have to manually add the tool. He described Python support as “fairly weak” at the time, though improving, and said the team had released around ten SDKs in the prior month with someone actively working on SDKs. When asked about LangChain, he said there was a basic SDK.

For export, Danny said many customers already have an OpenTelemetry stream and treat Raindrop as another target. They often want Raindrop to export the labeled signals, not necessarily own the raw trace source of truth. He said Raindrop supports exports to BigQuery and Snowflake with events and the signals classified for those events.

Historical analysis is supported in the described workflow. When asked whether a team could restore historical traces and later create a new signal to run postmortem analysis, Danny said users can ingest historical data and that, when a signal is created, Raindrop runs a quick backfill over the past couple of days.

The question of aggregated industry data—how many traces Raindrop processes, how many jailbreaks it sees across customers—was left unresolved. Ben said that would be interesting but Raindrop did not have anything like that available. Danny noted that there are mixed opinions about publishing such data and that customer compliance constraints may prevent it.

The substantive throughline was that production agent observability is not just logs plus traces plus offline evals. It is a signal system that combines objective telemetry with semantic detectors, experiment metadata, self-reports from agents, alerting, clustering, and trace inspection. The hard part is not only knowing when a tool threw an error. It is knowing when users are frustrated, when the agent silently worked around a constraint, when a new feature created a refusal pattern, and when an issue category exists that no one knew to test for.