Coding Agents Are Improving Their Harnesses, Not Their Models

For Ara Khan the practical case for recursive self-improvement begins with a narrower loop than an AI system designing its own successor. A coding agent can inspect failures and improve the harness around another agent: the tools, prompts and execution logic that shape whether it completes a task. Khan’s Cline demonstration showed that process operating on a benchmark. The model’s parameters were not what the system changed.

Khan described asking whether Cline could improve its score on Terminal-Bench 2.1. The benchmark contained 89 tasks. The baseline was 69 out of 89, or 77.5%. Khan said the model’s release paper reported a score of 85% or higher, prompting the team to investigate whether the model could do better with a different harness.

The work was split between two models: GPT-5.6 Sol led the improvement process, while Kimi K3 ran the tasks through Cline. GPT-5.6 Sol examined evaluation results, proposed changes to the harness and ran the benchmark again. Khan said the process ran for 17 hours without his intervention. It identified issues including loop detection, background processes that ran too long and file-index problems.

The chart reports other confirmation runs of 76 and 78 tasks, and a repeated mean of 77.5 out of 89, or 87.1%. It also notes that intermediate step heights are schematic, individual task contributions were not isolated, retry attribution was partly ambiguous, process guidance was not enforcement, and two interrupted runs were excluded.

The fixes shown were ordinary engineering: retrying after rate limits, checking whether output changes before flagging a loop, and keeping a worker alive when it exits prematurely. The significance, in Khan’s account, is the agent doing the laborious work of finding patterns in failures, proposing a change and testing it—not inventing a new way to code.

The improvement loop depends on evaluating agents through their work, not just their answers. A single-turn test might ask a model how many toes a cat has and grade its response. Khan contrasted that with asking an agent to set up an MCP server to connect to a database application. The agent may need to search documentation, read and edit files, and run commands. Its claim that it finished is not enough; the result must be checked against the task’s success criteria.

For this kind of task, the evaluation needs to inspect the outcome—the working setup—as well as the agent’s trajectory: the sequence of messages and tool calls leading to it. Khan’s evaluation diagram listed graders such as deterministic tests, an LLM rubric, state checks and tool-call checks. Tracked metrics included turns, tool calls, tokens and latency.

Khan described evaluation tasks running in separate sandboxes, with graders examining outcomes and aspects of the process. For Terminal-Bench 2.1, he said, Cline ran 89 tasks in 89 parallel containers. Each task ran independently. Graders could check whether it worked, how many turns and tool calls it took, how many tokens it used, how long it took and what trajectory the agent followed. Khan also emphasized guarding against shortcuts: an agent should not get credit for reward hacking, cheating or simply looking up a solution online.

An aggregate score can point to recurring failures. A file-reading tool might break on large files; inference might be so slow that a run times out; or an agent might lose focus or circle around a problem. The manual response is to inspect failed tasks, group them by type, make a small harness change, rerun the evaluation and repeat. Khan called this hill climbing “grunt-worky”: a run may take an hour, followed by the work of interpreting results and judging whether a proposed fix is legitimate and likely to generalize.

Automating that cycle makes it possible to run it for longer, but Khan’s slides also named fixed tests, preserved failures and human review as checks. A higher score needs an explanation: changes should apply beyond a particular task, rather than merely improve performance on it. Khan cautioned that making changes in real time could overemphasize the problem currently in front of the agent. He sees more promise in evaluating improvements across a wider range of tasks.

Khan defined recursive self-improvement as an AI system helping build a more capable successor, which repeats the process. His account of coding agents’ development offered context for why the idea now seems less remote: models moved from conversation to calling functions, then to using tools such as file editors and shell commands. Experiments with multiple agents communicating followed. Khan recalled those early multi-agent systems as “janky” and largely deterministic. The central shift, as he framed it, is the possibility that agents will keep getting better, faster and cheaper—and help improve the systems they use.

The Cline demonstration was one limited instance of that broader idea: a fixed model acted through a harness, while a second agent inspected outcomes and proposed changes to the harness. Khan described an inner loop, in which an agent interacts with a task and its environment, and an outer loop, in which a meta-agent learns across tasks and changes the system used on later ones. The diagram showed editable parts of the harness including prompts, tools, skills and context management. Updating model parameters appeared as an experimental path, not as part of the Cline demonstration.

Khan’s broader proposal is that improvement could extend across evaluation, training data, post-training, inference and interaction with the world. His slides pointed to examples including AI-curated pre-training data, Constitutional AI, model-generated evaluation scenarios and synthetic preferences used to improve an evaluator. These examples came from separate systems; the slide presented their connection as a framework for a larger loop, rather than a single integrated process.

Inference offers another example. Khan cited an OpenAI-reported account in which GPT-5.6 Sol rewrote and optimized production kernels—the code that performs a model’s mathematical operations. The slide attributed a 20% reduction in end-to-end serving costs to those efforts combined with broader kernel advancements from GPT-5.6 Sol. It also reported more than 15% higher token-generation efficiency and noted verification tooling, including FpSan, to help check the kernels’ correctness. Khan presented this as a case of models helping improve the machinery used to serve models.

He also showed a post by Mark Saroufim quoting a GPU-kernel competition participant who said their entry was entirely AI-generated and that they had never hand-written a GPU operator. Khan pointed to the example as evidence of agents contributing to kernel optimization and the prospect of faster, cheaper inference.

Khan argued that software is not the limit of the idea. A physical system can also provide a cycle of action, observation and correction. He showed a robot typing on a laptop and raised the possibility that a robot could interpret the results of its own attempts to solve robotics problems and improve from them.

A more concrete experiment involved a small device Khan had with a camera, speaker and microphone. He found that it responded slowly, connected it to a computer and used coding agents to run tests against the hardware. The tests measured end-to-end latency and helped identify slow responses and algorithmic problems. Khan said he then set up overnight experiments in which agents could update the firmware, tool calls, inference model and speech-to-text model. He described the device as substantially optimized through that process.

The common feature is that the system can try something and measure the result. In software, the check might be a test suite; with hardware, it might be latency or another observable response. Khan also described human corrections as potential training signals. If an engineer corrects a model’s approach to configuring a cache or setting a time-to-live, the correction can record what changed and why. In his framing, post-training uses such experience to push the boundary of what a model can reliably do.

The loops could reinforce one another: cheaper inference could allow more experiments within a fixed budget; those experiments could produce better data and models; and better models could help improve inference and evaluation. Khan said parts of this loop are already real. His examples ranged from benchmark-driven harness changes to hardware experiments.

Khan’s career advice follows from the distinction between work that benefits from better models and work that stronger models may simply make unnecessary. A side project that a frontier coding agent can produce in one shot may no longer demonstrate much about its author’s engineering ability. Some narrow applications, he argued, could be one model release away from being wiped out. He also questioned whether repeatedly asking an agent to do routine tasks is the most intellectually demanding use of an engineer’s time.

Khan recommends moving closer to the improvement loop: work on pipelines, inference, post-training, evaluation, physical interaction, hardware or networking—areas where measurable progress can make further improvement possible. His advice is to find work where improvements in models create leverage, while the problem still requires contributions that a model release will not make irrelevant.

He highlighted a slide attributed to Sasha at Anthropic, framed around the question of what to do if one has only one to five years to contribute intellectually. The slide advised preferring ambitious, targeted, fast collaborations and keeping a project’s timescale shorter than the characteristic timescale of rapid model progress. Its warning was that slow, meandering work might be overtaken: if a newer foundation model is likely to do the work better and faster before a project is finished, it may make sense to wait rather than undertake that project now.

The slide also invokes the “bitter lesson”: general methods that leverage computation tend to become more effective, so projects should benefit from scale. Khan emphasized avoiding projects that will be solved by scale alone before the work is complete. The aim is an endeavor that gets more valuable as models improve without being fully displaced by the next release—and, ideally, one that contributes to the improvement pipeline itself.

Khan points to data pipelines and physical hardware as areas attracting work and investment, and asks engineers to examine whether their own day-to-day contribution is being compressed by agents. If a model is already one-shotting the problems in front of someone, he suggests stepping back to look for a bottleneck where that person’s judgment and ability can still change the result.