Agent Skills Need Evals Before They Reach Customers

Philipp Schmid draws a sharp distinction between the coding agents engineers use themselves and the agents they put in front of customers. An engineer using Cursor, Claude Code, Codex, or another coding environment can notice that a capability was missed, stop the run, reprompt, or invoke a slash command directly. That human recovery loop masks failures.

A customer-facing support bot or workflow agent has no such protection. The user does not know that a refund skill, deployment skill, or document-processing skill exists, much less how to trigger it. If the model fails to invoke the right capability, invokes the wrong one, or follows misleading instructions, the customer simply sees a broken agent. The further the end user is from the skill system, Schmid argues, the higher the reliability bar becomes.

The problem is not merely that skills sometimes crash. Bad skills can quietly corrupt an otherwise plausible output, making failure difficult to detect through casual manual testing. Schmid points to SkillsBench’s indexing of more than 47,000 skills across 6,300 repositories, where almost none had tests. In his account, many are AI-written, lightly inspected, tried manually once or twice, and then shipped.

That process is particularly weak for nondeterministic systems. A failed task may mean the skill is poorly written, that it did not trigger, that the model took an unhelpful path, or that the task itself exceeds the model’s capability. Without a repeatable evaluation harness, those causes are easy to conflate.

A skill, in Schmid’s formulation, is not a retrained model feature. It is a versionable folder organized around a required SKILL.md file, with supporting scripts, reference documents, templates, specifications, or other assets. Its design relies on progressive disclosure: frontmatter supplies a name and description for routing; the SKILL.md body provides core instructions after activation; and references or scripts hold material the agent can read or execute only when needed.

A skill has to earn the context, latency, and maintenance it adds. The useful measure is whether it improves a defined user outcome.

Schmid’s case is not that skills are inherently suspect. He cites SkillsBench 1.1 results displayed during the talk: curated, human-written skills increased overall task-resolution accuracy from 33.9% to 50.5%, a gain of 16.6 percentage points across the evaluated settings.

But the same evidence makes a narrower point: adding instructions is not equivalent to adding capability. The displayed comparison found self-generated skills worse than the base agent by 8.1 to 11.5 points, while curated skills performed substantially better. Schmid’s warning is aimed at a common workflow: ask a coding agent to generate a SKILL.md, glance at the result, see that it broadly resembles the intended guidance, and accept it. That is not a validation process.

Long instructions consume room the model could use for the task. The description is present on every call; a long SKILL.md enters context whenever the skill activates. Material that is irrelevant or diffuse can bury the instruction that would actually change behavior. Schmid’s practical threshold is simple: a skill above 500 lines merits close inspection.

He separates skills into two categories with different expected lifetimes. Capability skills address something the model cannot yet do consistently: a new API, an internal system, a schema, a framework migration pattern, or a specialized parsing task. They are temporary by design. As models improve, an eval can reveal that the capability is now native and the skill should be removed.

Preference skills, by contrast, encode a team’s own workflows, conventions, deployment rules, review practices, formatting requirements, or Git patterns. These tend to be durable because a general-purpose foundation model has no reason to absorb a company’s local process. Here, evals protect against regressions when the model or agent environment changes.

For model-invoked skills, the frontmatter description is effectively the routing mechanism. The model sees that concise description and decides whether to load the skill. Schmid says trigger problems account for more than half of skill failures in his experience: a skill may fail to fire when needed, or hijack unrelated requests when its scope is too broad.

A description such as “Helps with documents” or “API helper” gives the model little usable basis for a decision. A stronger description states both the capability and the trigger context: for example, creating, editing, and analyzing .docx files, including tracked changes, comments, formatting, and text extraction.

The boundary matters as much as the intended use. A React-component skill should say that it applies to React components and, where useful, that it should not trigger for plain HTML/CSS or backend routing. Testing only prompts where a skill ought to fire can optimize it toward overreach; negative and near-miss cases expose false-positive triggering.

There is also a design choice between model-invoked and user-invoked skills. A user-invoked skill, such as a slash-command workflow for creating a pull request or cleaning a worktree, imposes no ambient context cost until someone explicitly calls it. Schmid considers these underused for intentional, repeatable developer workflows.

Model-invoked skills are necessary when an end user cannot be expected to know the workflow or name the capability. But their trigger behavior needs explicit evaluation: does the skill activate for a relevant prompt, and stay inactive for an adjacent but irrelevant one?

That is distinct from the ultimate evaluation. Schmid recommends grading whether the user’s task was completed correctly, not requiring that the agent load a skill on its first turn or follow a specified sequence. A skill can trigger later than expected—or the model can solve a task without it—and still produce the right outcome. Trigger checks diagnose routing; outcome checks establish whether the system served the user.

The central writing rule is to use directives rather than essays. “The interactions API is recommended for multi-turn chat because it handles session state automatically” leaves the model to infer the desired behavior. “Always use client.interactions.create() for chat. Never use the legacy generate_content API” is explicit about both the action and the prohibited alternative.

Schmid does not argue for a rigid procedural script in every skill. He warns against filling skill files with step-by-step instructions for tasks where the model needs room to adapt. A skill should describe the intended outcome and its constraints: update the database port in config.json, ensure the file parses correctly, and run tests before opening a pull request. The agent can decide how to reach that state and recover if one route fails.

If a workflow truly must be followed in an exact, deterministic sequence, Schmid’s advice is not to write a more prescriptive skill. Write a script and have the agent run it. A model does not need to spend reasoning tokens reenacting a fixed mechanical procedure.

The same test applies to no-ops: decorative phrases that do not change agent behavior. AI-generated skills often include instructions such as “Be thorough,” “Write clean, high-quality code,” or “Make implementation easy to read.” Schmid’s test is empirical: remove the line. If output does not change under evaluation, delete it.

Detailed material still has a place, but it should be available on demand. Keep frontmatter minimal, keep core instructions concise, and move specialized documentation into reference files. A multi-cloud deployment skill, for example, can point to separate AWS, Google Cloud, or Azure references rather than loading all three procedures into every activation.

Schmid’s practical example was a skill for Gemini’s Interactions API, released after the relevant Gemini models’ training. The intended behavior was specific: use the correct SDK imports, use the Interactions API, reject deprecated model names, and generate current code across Python and TypeScript.

The displayed case study used 117 test cases spanning chatbots, streaming, document processing, and SDK features. The slide reported with-skill gains from 28.2% to 96.6% for Gemini 1.5 Pro, from 6.8% to 93.2% for Gemini 2.0 Pro, and from 6.8% to 89.7% for Gemini 2.0 Flash. Schmid described the end result more generally as reaching nearly 90% valid Interactions API code.

The harness itself was deliberately simple:

• Start with a JSON or YAML prompt set: happy-path prompts, negative or near-miss prompts, and real production traces where available. Each case can specify the language, whether the skill should trigger, and the checks expected of the output.
• Run the target agent programmatically in a clean workspace. The workspace can install required packages and create task files, but should not contain prior traces or artifacts that let the agent succeed from leaked context.
• Use deterministic validators first. Schmid says most skill checks can be regex assertions against extracted code: correct SDK imports, prohibited deprecated model IDs, required methods, or old API patterns.
• Use LLM-as-a-judge selectively for requirements regex cannot settle, such as code quality, design choices, qualitative instruction adherence, or an agent’s full trace. Give the judge the original prompt, generated artifact, and an explicit rubric; require structured pass/fail output.
• Repeat cases several times and compare runs with and without the skill.

The low-cost validators are what make frequent testing practical. Schmid describes checks for imports, model names, methods, and legacy patterns as basic assertions that can run cheaply and repeatedly, without using an LLM judge for every output. Model-based judging belongs where qualitative evaluation is actually needed, rather than where deterministic checks can settle the question in milliseconds.

He recommends starting with 10 to 20 real prompts, though even five cases can expose failures: a small happy-path set, a small negative set, and production-derived cases where available. Every user-reported issue should become a regression case rather than a one-off debugging event.

At Google DeepMind, Schmid says, skills carry evaluations alongside them, and every change to a skill runs those evaluations. A change does not merge unless it improves the test cases, or adds new evals. The internal implementation is not the YAML example shown in the talk, but the model is similar: test cases define prompts, environments, setup actions, validators, and, where needed, judge criteria.

Isolation prevents a misleading kind of success. An agent running in an existing environment may find prior conversations, artifacts, executions, or local files that provide the answer or reveal the skill’s content without requiring the tested behavior. A clean workspace makes the result more attributable to the skill and agent configuration under evaluation.

One successful run is also insufficient. Agent behavior is probabilistic, so Schmid recommends three to six trials per case and assessing the pass-rate distribution rather than celebrating the best observed result. He also recommends testing across the harnesses people actually use. A skill that works with one model or agent framework may behave differently in another, including environments such as Gemini CLI, Claude Code, or Cursor.

The final question for a capability skill is whether it still earns its place. Run the same suite with and without the skill loaded. If performance is materially higher with the skill, it remains useful. If the base model matches its pass rate without loading it, the model has absorbed the capability and the skill can be retired—saving context, latency, maintenance, and token cost.

Retiring a skill does not require retiring its eval. Schmid recommends preserving that test as a regression check. If base-model behavior later degrades, the existing evaluation can expose the loss and support a decision to reintroduce a skill or adjust another part of the agent system.