Agent Traces Turn Hidden Workarounds Into Fixes for Skills and Configuration

A software factory automates the software development lifecycle: agents take work from a task list, board, or Slack message through triage, specification, implementation, verification, review, shipping, and monitoring. But automating that cycle is only part of the engineering problem. The system that directs the agents also needs to improve.

? ben-holmes calls that work factory engineering: improving the skills, configuration, and architecture that shape how agents operate. His focus is narrower than the whole factory. It is a feedback loop in which agents review their past work, score it against explicit criteria, and use repeated failures to propose changes to their setup.

The proposed loop has four steps: review conversations, score their quality, isolate recurring failures, and propose improvements. In a team environment, the scoring can run after work is completed, while a separate agent periodically reviews the failures for patterns. Holmes describes a daily cycle: gather reports from the previous day, identify common causes, and give an improvement agent access to the relevant skills repository and codebase. That agent can then suggest a configuration or skill change, potentially as a pull request.

This is not just a record of whether a task succeeded. An agent can deliver working code while wasting time, ignoring preferred testing practices, or struggling with instructions it eventually works around. Inspecting the process can reveal those costs.

For the loop to work, the team needs its agent conversations in one place. ? ben-holmes says local tools such as Claude Code or Codex store conversations on a user’s machine, which can be enough to examine one person’s setup. A team could collect those files centrally—for example, he has spoken with teams that copy users’ .claude folders into a shared S3 bucket. In a software factory running tasks on shared infrastructure or isolated cloud sandboxes, centralizing traces may be part of the setup already.

A useful trace contains three kinds of information: the transcript, the tool calls, and the output. The transcript records what the human and agent said. Tool and MCP calls, along with file-system interactions, show what the agent did outside the conversation. The output might be the code diff or the pull request produced by the run.

That distinction matters because agents may work around failed instructions without reporting the detour. As Holmes puts it:

He describes agents trying alternatives after instructions fail, then reporting the result without surfacing the detour. If the trace contains only the conversation, a team can miss the failed calls and repeated attempts that explain why a task took longer than expected.

Once the traces are available, an LLM judge can grade them against criteria the team defines. ? ben-holmes’s examples are redundant tests, verbosity, and efficiency. These are areas where a team may have a recognizable problem and can state what better performance would look like.

For redundant tests, a team might judge whether an agent added more tests than the project’s guidance calls for. Holmes gives the example of a team that prefers end-to-end tests but sees an agent write many unit tests instead. Whether that is a failure depends on the team’s testing preferences, which is why the team must encode its own standard in the rubric.

Verbosity covers how the agent communicates as well as how it writes code. Excessive comments or long summaries can make a change harder to review. Holmes connects this to the practical cost of code review: if agent-written explanations and comments obscure the change, they can slow down the human review the workflow still depends on.

Efficiency requires the tool-call history. In Holmes’s example, instructions for an end-to-end test environment were out of date. Rather than identify the problem, the agent tried different port numbers and eventually got the work done. From the outside, the task merely took longer than expected. In the trace, the team could see the avoidable exploration—and identify that better instructions might prevent it.

Holmes presents the rubrics as Markdown files, with a name, a description, labels for different levels of performance, and a short list of factors to assess. The efficiency rubric shown in the talk asks whether the agent worked directly toward the result or wasted effort on redundant steps, avoidable rework, or unnecessary back-and-forth. Its assessment prompts include rework from mistakes, cost to the human, information gathering, routine-step overhead, batching, and verification timing.

The rubric should be explicit, but not elaborate for its own sake. Holmes compares it to a teacher’s grading criteria: define what counts as success and failure clearly enough that a judge can apply the standard. A team can begin with a few bullet points and refine them. He describes rubric design as more art than science.

The grade also needs a threshold. In the example he gives, low scores such as 0.2 or 0.4 fall below the team’s bar; scores above it count as passing and may not need further investigation. The pass-or-fail result follows the rubric and threshold the team has set.

? ben-holmes demonstrates grading an illustrative conversation in which an agent implemented an auth service. A grading agent reviews the session against separate criteria, rather than assigning one undifferentiated quality score.

On redundant tests, the agent fails: it wrote many unit tests even though the team’s guidance favored end-to-end coverage. On verbosity, it passes: the pull request is clear, and the agent did not add too many comments. On efficiency, it fails: it spent time exploring the test environment when it should have had upfront instructions for how to use it.

In this example, the failures point to different issues: the testing guidance and the test-environment instructions. The verbosity score passes, so that criterion does not flag a problem in this conversation.

A team can grade every conversation or sample a subset to manage token costs. Holmes describes one agent run per conversation as an option, with a PR merge as a possible trigger for grading. Each report can include not just a pass or fail, but an explanation of why the judge assigned it. Those explanations become input for looking across conversations for repeated causes, rather than treating each grade as an isolated incident.

? ben-holmes says a daily improvement agent can receive failed reports and look for commonalities. In one example, five conversations had written redundant tests. The proposed response was to add a “coverage pyramid” to the test-writing skill, clarifying when to use unit versus integration tests.

The improvement agent receives access to the skills repository and codebase so it can suggest changes to the shared setup. Holmes describes suggestions such as skill edits or factory configuration changes. In the Warp example later in the talk, the suggestions appear as pull requests in an inbox for the team.

For people working with a single-user setup, Holmes presents Skill Doctor as a way to review conversation histories from tools including Claude and Codex. In the demo, scoring agents worked in batches across Codex sessions, evaluating code quality and efficiency. The resulting report card covered 12 conversations and displayed an overall grade of B.

The report’s proposed changes drew on the explanations behind the scores. In one case, the agent repeatedly struggled to find a port number in its test environment. Skill Doctor proposed updating the end-to-end testing instructions so the agent would find a port before trying to use one. In another, it noticed repeated back-and-forth over changelog wording and proposed making the changelog skill more prescriptive.

Holmes describes Skill Doctor’s one-time review as a way to see what might be improved before setting up continuous scoring. Teams can start with a small rubric and a sample of conversations; sampling can also limit token costs. He says the rubric can begin with a few bullet points and be built up as the team learns what it needs to assess.

A continuous loop needs a way to tell whether a proposed change helped. ? ben-holmes argues for tracking scores over time and comparing changes in those scores with changes merged into the agent setup. If an efficiency score rises after a skill change, a team has a visible signal to investigate. Holmes describes the resulting trend as a way to see whether a change coincided with a shift in performance; the trend is evidence to inspect, not by itself proof that the change caused the shift.

This visibility is also a contrast with less explicit forms of agent memory. Holmes says that systems in which agents reflect on conversations overnight can be useful, but it may be difficult to see what the agent concluded or retained. Scores and a dashboard offer a more legible view of how conversations performed.

In Warp’s factory setup, Holmes showed an inbox of self-improvement pull requests across the agents supporting the team. One merged change addressed a problem in the triage skill: the agent lacked configuration to spawn research subagents and repeatedly tried to work out how to do so. It proposed adding the missing configuration.