Private Evals Determine When Agent Harnesses Need Customization

Harrison Chase defines an agent as three things: a model, the context supplied to it, and a harness that orchestrates the two. Owning an organization’s intelligence, in his framing, means retaining control over all three.

Model ownership includes the ability to switch models—not only to avoid lock-in, but to use the strongest available option. Context ownership covers the material that makes an agent useful in a particular organization: prior conversations and memory, semantic knowledge retrieved for a task, and procedural knowledge such as skills. But Chase’s focus is the harness: the control layer that gathers fixed and dynamic context, puts it into a model’s context window, receives a response, and determines what happens next.

At its simplest, the architecture is an LLM calling tools in a loop. A request arrives; the model produces a response that may include an action; the harness invokes the relevant tool; the resulting observation goes back to the model; and the loop continues until there is a result. Chase said this simple but general architecture sits behind “pretty much every agent” in use today.

The loop needs to do considerably more than call a tool. An agent may need an execution environment in which it can act, connections to company data, a way to manage context across long runs, parallel work across tasks, human intervention, and mechanisms for improvement. Interactions with external systems produce new observations, which become more context for subsequent model calls. The harness coordinates that growing system.

That definition shifts the emphasis away from the model call in isolation. A bad output may reflect model capability, but it may also reflect the sequence of actions, tool outputs, prior messages, and retrieval choices that produced the model’s actual input.

The core loop and the modifications placed around it are distinct. Harrison Chase describes a minimal harness that can be extended with middleware at defined points: before the agent begins, before a model call, around a model call, around a tool call, after the model responds, and after the agent completes. Coding-agent systems often describe the same pattern as hooks or plugins.

This approach allows significant changes without discarding the basic model-tool-observation cycle. A harness can add a code interpreter, sandbox, filesystem access, planning and sub-agents, skills, memory, human-in-the-loop controls, prompt caching, and context-management mechanisms.

Summarization illustrates the pattern. Before invoking the model, the harness can check whether accumulated context is too long and summarize it before the next call. Context offloading is another example: a wrapper around a tool call can move large tool outputs out of the active context rather than continually carrying them forward. The agent remains a loop; the harness changes the conditions under which the loop operates.

Chase presents DeepAgents as a more general-purpose, model-agnostic harness built on a minimal LangChain core. Its additions—filesystem connections, skills, sub-agents, and related controls—are not a different underlying agent theory. They are interventions at leverage points in that loop.

There is another, more explicit form of customization: a cognitive architecture. These systems lay out prescribed stages such as search, planning, execution, and validation, or generate research questions, fan them out, and assemble a report. Chase says such bespoke architectures were more common in 2023 and 2024, when models were less able to operate effectively through a general tool-use loop. Some of that work has now moved into the harness itself.

Still, explicit architectures remain useful for tightly specified flows. Chase’s recommendation is not that every application should begin with a custom graph of stages and gates. Start with a general harness for the faster path to value, then add checks, controls, and more directed flows as the use case becomes clearer and the need to excel on it becomes more specific.

The decision to build a custom harness is a spectrum shaped primarily by how close the task is to the distributions on which a model has been trained and reinforced. As Harrison Chase puts it:

For tasks that are highly in-distribution, Chase recommends beginning with an off-the-shelf harness—such as Claude Code, the Claude Agent SDK, Codex, DeepAgents, or another general-purpose option. At the far end of the spectrum, an organization may need a fully custom cognitive architecture.

But a broad task can be out of distribution while containing narrower operations that are familiar to the model. Legal AI is Chase’s example. The overall legal workflow may warrant a custom harness, yet file editing is an operation that major models have been reinforced to perform. Crucially, they have been trained to edit files in different ways. OpenAI and Anthropic models, he says, are each best at editing files through the patterns used in their respective harnesses.

The implication is not merely “customize the harness.” It is to customize selectively. DeepAgents uses model profiles that switch file-editing implementations depending on the model in use. That preserves a model-specific, in-distribution interaction inside a broader harness tailored to an unfamiliar domain. A custom legal agent, in this formulation, should not discard a model’s learned editing behavior merely because the larger workflow is bespoke.

The same spectrum applies for reasons beyond task fit. Some financial-services customers, Chase says, reject a broad, autonomous agent because it feels too unpredictable. They prefer a more controlled cognitive architecture with explicit steps. Middleware and hooks offer a middle ground between an unmodified general-purpose harness and a wholly bespoke system.

Chase does not offer a confident forecast of whether harnesses will ultimately converge. General-purpose harnesses have become good enough for many basic starting points, and labs’ coding systems may converge if they continue optimizing for coding. Yet small differences already matter—particularly in file editing—and future specialization could push systems apart. If one lab develops especially capable biology-oriented agents, Chase suggested, its harnesses could become specialized for that work rather than generic coding.

The build-versus-buy decision therefore has to be evaluated on a team’s own tasks: which model-harness pairing succeeds more often, at what latency and token cost, and with what degree of predictability and control?

Evaluation is what converts a harness decision from a framework preference into a measurable choice. Harrison Chase argues that a team able to compare model-harness combinations against its own tasks can see which changes improve results, which introduce regressions, and whether better accuracy comes at an unacceptable latency or token cost.

He highlighted three propositions displayed from a Satya Nadella post: organizations should create private evals because evals define what “good” means internally; retain ownership of memory, traces, feedback, decisions, and institutional context; and build a continuous learning loop through which AI investments compound.

For Chase, every company deploying a mission-critical agent—whether internal or customer-facing—will build a benchmark for it. The benchmark has two functions. It detects regressions when a model, harness, prompt, or context strategy changes. And it provides a surface on which a team can “hill climb,” iterating toward better performance by modifying the harness or model.

He described Harbor, an open-source eval runner created by the makers of SWE-bench 2, as “becoming” an industry standard for defining these benchmarks. Its model is straightforward: run an agent against a dataset of tasks, typically in sandboxes so tasks can be isolated and parallelized.

The verifier can run code or unit tests, use another LLM as a judge, or use an agent as a judge. A benchmark built this way lets a team run the same task set against alternative agent and model configurations, then score the results according to its own definition of success.

The frontier-bench leaderboard Chase displayed makes the comparison tangible: it lists agent-model configurations alongside reasoning effort, accuracy, release date, organizations, and token use. Its leading displayed configuration, mini-SWE-agent with Opus 5 at maximum reasoning effort, showed 43.5% accuracy, while other listed configurations ranged down to 15.3%.

Accuracy is not enough. In the LangSmith evaluation view Chase demonstrated, experiments could be compared using a reward or feedback score along with latency and token consumption. The question is not simply which configuration wins a benchmark, but whether the improvement is sufficient for the deployment’s time and cost constraints.

Observability is often treated as a basic operational feature, but Harrison Chase considers it underrated in agent systems. His diagnostic model is simple: when an agent fails, an LLM call has gone wrong. That can happen because the LLM is not capable enough, or because the context it received was inadequate. He believes the latter is more commonly the cause.

To diagnose the difference, teams need the exact model inputs and outputs and the trajectory that produced them: which steps ran, which tools were invoked, what observations came back, and how those observations accumulated in the context window.

A trajectory view can make this legible by presenting the sequence of human messages, model responses, and tool activity while collapsing some low-level detail. Chase compares it to the message stream visible in coding agents: a person supplies a request, the agent makes tool calls, the results return as messages under the hood, and the agent responds.

That trajectory is important, but not sufficient. The debugging view must also allow an operator to inspect individual calls and see what occurred inside the model interaction. Without that record, a team cannot determine whether to change a prompt, retrieval behavior, memory, tool handling, model choice, or the agent’s broader control flow.

Continuous improvement begins with a straightforward loop: build a first version of the agent, run it, collect traces, curate the data, and run experiments on the curated set. Harrison Chase emphasizes that the complexity lies in deciding which traces expose generalizable failures rather than isolated incidents.

Feedback is the crucial input to that curation. Chase argues that thoughtful UX can elicit useful feedback without relying on explicit thumbs-up or thumbs-down signals. Teams can supplement it with synthetic feedback through online evaluators that inspect traces. Those evaluators may be smaller language models, off-the-shelf models used with custom prompts, or conventional code when the property to test is simple enough.

Cost constrains this work. Running a high-end model as judge over every incoming trace can become expensive, Chase says. LangChain has fine-tuned smaller language models for some evaluation tasks as a cheaper, faster alternative.

The resulting data can improve any of the three agent components. Harness engineering changes the loop, tool handling, and controls; fine-tuning changes the model; memory work changes the context. LangChain concentrates on the harness portion, but Chase places it within the same overall feedback system.

LangSmith Engine is LangChain’s attempt to automate some of the trace-curation work. Chase describes it as an agent that searches production traces, identifies recurring issues, collects linked evidence, and proposes changes to context, instructions, or harness behavior.

The demo centered on a documentation agent that repeatedly generated Python code using the JavaScript-style camelCase parameter systemPrompt, although the relevant Python factory expected system_prompt. The Engine issue linked 11 traces and described the agent repeating the same broken keyword argument through follow-up attempts. Its proposed fixes included changes to prompt context and instructions, along with a harness-level addition.

LangChain also runs Engine on its own operation and maintains Issue Bench, a Harbor-formatted benchmark for comparing models and harnesses on this issue-identification work. Chase says that, in one comparison, CodeAct performed especially well because it aggressively wrote small scripts to analyze traces. LangChain subsequently ran a sprint to bring that behavior into Engine’s core harness, a change he called its “Codex-ification.”

The example is the improvement loop in miniature: compare alternative harness behaviors against a benchmark, identify what produces stronger results, and incorporate that behavior into the production system.