Coding Models Optimize for Tests, Not Maintainable Architecture

? dexter-horthy rejects the premise that teams can eliminate code review by wrapping coding models in better loops, more agents, larger token budgets, or more elaborate harnesses. Those techniques can improve execution, he argues, but they do not solve the central problem: today’s models are not trained to preserve and improve a codebase’s maintainability over time.

The appeal of the “lights-off” software factory is obvious. An agent takes work from an issue queue, writes code, runs tests, opens a pull request, and perhaps sends failures, incidents, and user feedback directly back into the same system. Human beings stop reading code; their job becomes deciding how much demand to put into the queue. Horthy cites StrongDM’s publicly described “dark factory” model, where code is neither written nor reviewed by humans and token spending becomes the route to leverage.

His objection is not that coding agents are useless, nor that developers should abandon loops. He distinguishes the situation from hobbyist “vibe coding,” where an imperfect side project may have few durable consequences. The relevant setting is a complex, evolving system: the sort of codebase usually called brownfield. In fact, he argues that agents can begin to struggle much sooner than the conventional image of a decade-old enterprise system suggests—after only three to six months of rapid AI-assisted shipping.

The warning signs, in his account, are already visible. He points to reports of agent-related production mishaps and presents Faros AI figures from an analysis of 22,000 developers across 4,000 teams. Faros associates high AI adoption with deteriorating review and production-quality measures.

The usual response, Horthy says, is to blame the operator: teams are “holding it wrong,” need better context, should spend more tokens, or should add an adversarial review agent. That view promises the best of both worlds—10 to 100 times more software, high quality, and no human code review. Horthy’s claim is more restrictive: harness engineering can make a model more effective at what it has been trained to do, but it cannot substitute for a reward signal that does not exist.

The practical implication is not that every task deserves a design process. Horthy explicitly leaves room for small work to go straight to an agent. The problem begins when teams treat consequential changes to a shared system as though a passing harness is an adequate replacement for agreeing on behavior, boundaries, implementation shape, and validation before code is generated.

? dexter-horthy locates the distinction in maintainability. He uses Martin Fowler’s “shotgun surgery” code smell as shorthand: a small change becomes difficult because it requires touching many unrelated places, each with a chance of breaking something else. A codebase may pass its current tests and still become increasingly hard to change safely.

Horthy’s own experience convinced him this was not a theoretical concern. In July 2025, HumanLayer tried operating with the lights off. After several months, an issue emerged that the agent could not resolve despite research, reproductions, and increasingly elaborate prompts. The team had to dig through a codebase it had stopped reading. Meanwhile, the site was down, users were angry, and Horthy was confronting what he calls “slop code” that had accumulated without sustained human steering.

That outcome is consistent with a gap Horthy sees in model progress. In his view, models have improved substantially at solving discrete problems—fixing a bug, generating a feature, producing a new marketing site—but have made little comparable progress in their ability to improve a codebase’s overall quality across a long sequence of changes. He acknowledges that there are no good benchmarks that prove this claim directly. Horthy says practitioners who have used coding agents over time often recognise the pattern of a system becoming harder to work in.

The reason, he argues, is visible in how coding agents are trained. A typical reinforcement-learning setup generates many possible trajectories for a task, grades the resulting work, and makes successful trajectories more likely in future model behavior. The model receives a problem, searches a repository, edits files, runs commands and tests, and eventually produces a patch. The training system needs a score.

SWE-bench Multilingual illustrates the structure. Its tasks are drawn from open-source repositories such as Redis, jq, and Django and are intended to take roughly 15 minutes. The agent starts from the commit before a human fix. It receives a real issue description, while the human patch and added tests are hidden. At grading time, the system captures the agent’s patch, discards any changes it made to tests, applies the hidden test patch, and checks two conditions: existing behavior still works, and the new intended behavior works.

Horthy’s Fastlane example is simple. A Ruby action crashes when optional include and exclude parameters are omitted because the code calls a method on nil. The human fix gives both values empty-array defaults. An agent earns the reward if its patch preserves old tests and passes the hidden test covering the new behavior.

That is a powerful training signal for correctness in a bounded task. But Horthy’s point is that the reward says nothing about whether the agent’s solution improved or eroded the system’s design. A model can learn to get to green by adding unnecessary error handling, using broad casts, bypassing type constraints, or making a localized patch that increases future coupling. He illustrates the incentive with a try/catch introduced merely to survive JSON parsing and a TypeScript value cast through unknown to any so a new property can be attached.

The model is therefore not irrationally producing poor structure, in Horthy’s view. It is optimizing what the training setup can score. The cost of architecture is especially difficult to turn into a reinforcement signal because it arrives late. A test can provide feedback minutes after a patch is written. The consequences of poor boundaries, duplicated logic, unclear ownership, or badly chosen abstractions may not emerge until months later, when a seemingly modest feature requires changes across multiple systems. By then, Horthy says, it is hard to propagate a useful reward back across the many intervening agent actions.

? dexter-horthy uses Claude Code’s rise to distinguish a better harness from a solution to maintainability. Before Claude Code, command-line agents such as Aider and Codebuff already exposed the familiar tool set: read, write, edit, search, and shell access. The interface alone, he argues, was not the differentiator.

His explanation is that Claude Code was the first case in which a model lab trained a model against the harness it intended to distribute. The resulting model became highly capable at operating the tool loop: reading context, choosing a next action, searching, editing, executing commands, and iterating toward a working patch. Horthy cites a related point from OpenAI’s Codex team: a harness builder that does not own model weights and cannot reinforce learning within its own environment will remain disadvantaged against an organization that owns both model and harness.

That advantage is real, but bounded. Training against a particular environment can make a model much better at completing verifiable tasks in that environment. It does not necessarily give the model a reliable account of which abstractions, boundaries, dependencies, or program layouts will keep a codebase workable after months of change.

Better evaluation work is emerging, Horthy says. SWE-Marathon from Abundant AI uses tasks ranging from four to 400 hours and includes projects as broad as recreating Excel-like functionality, with an 18-channel behavioral reward. DataCurve’s DeepSWE uses novel, longer tasks on open-source repositories rather than relying on issues already solved in the real world. Cognition’s Frontier Code evaluates multi-pull-request work and adds mechanisms such as penalizing tests that would not fail against the pre-patch code; it also uses a judge model to assess explicit quality rules.

These are meaningful attempts to broaden what gets verified. But Horthy does not think model-based quality judgment resolves the problem entirely. If a model fully knew what good code looked like in the relevant context, he says, it would be more likely to produce that code in the first place. Review agents and additional inference-time effort can raise the floor, but they remain constrained by what can be taught and reliably rewarded.

For now, his conclusion is blunt: teams remain responsible for reading the code they ship. The relevant question is how to make that responsibility inexpensive enough that it does not erase the speed gained from agents.

Horthy’s alternative is not a return to slow, manual implementation. It is a change in where teams spend attention. Conventional software processes already use product discussions, architecture proposals, and sprint planning partly to reduce rework: the team aligns on intent before a reviewer has to reconstruct it from a surprising pull request.

AI can compress implementation without making that alignment obsolete. Horthy’s “lights-on” factory restores human code review, then reduces its burden through structured pre-planning. The operating model separates small changes from work that changes a system’s behavior or shape.

Small work can still go directly to an agent. For larger or consequential work, Horthy recommends four increasingly concrete stages before implementation.

First is product review: identify the problem, intended behavior, success criteria, and, where useful, visual mockups. His example product document concerns “context shards,” reusable instructions and facts gathered from previous agent sessions. It identifies a concrete failure mode: users repeatedly supply the same steering, while lessons learned by one teammate do not become shared guidance. Its success measures include fewer corrective prompts and adoption of useful shared instructions.

Second is system architecture. Teams establish component contracts, data models, boundaries, and constraints before code generation begins. Horthy’s adoption-dashboard example specifies a contract-to-API-to-UI path, identifies its data sources, and makes the route read-only rather than adding writes, migrations, or new tables. The point is not the dashboard itself. Before implementation is delegated, the team has decided what the new surface does, which systems it touches, and what it deliberately does not change.

Third is program design, which Horthy considers underemphasized in agentic coding. Architecture alone is not enough for a model to “cook,” he says. Teams should make the program’s shape explicit through types, method signatures, module layout, call stacks, component trees, and dependency boundaries. He points to Cloudflare engineer Dillon Mulroy’s use of final call graphs in implementation plans: a concrete map of how work moves through a system can guide both the agent and the eventual reviewer.

Fourth is vertical slicing: agree on the order of implementation, incremental pull requests, dependencies across a monorepo, and validation at each phase. Horthy contrasts this with “horizontal” plans that complete one layer of the stack at a time. A vertical slice instead cuts through frontend, API, services, and database in a narrow, testable increment, with validation agreed in advance.

The planning is not bureaucracy for its own sake. Horthy’s operating claim is that roughly 30 minutes of pre-planning and alignment can remove hours of later review and rework.

Horthy pushes back on the complaint that AI creates too many pull requests to review. The deeper issue, he says, is too many bad pull requests: changes that arrive without a clear shared understanding of what they are supposed to do, how they fit the system, or how they should be validated.

A well-planned pull request gives the reviewer something specific to confirm. The problem and desired behavior should already be clear. The reviewer should be able to see that the implementation respects the agreed component contracts and program shape, that the change is confined to the intended slice, and that the expected checks have been run. In Horthy’s formulation, the reaction should be: this is what we discussed; this is the structure we agreed on.

The code still deserves scrutiny, but the reviewer is no longer forced to reverse-engineer product requirements and architecture while also searching for defects. That distinction matters because even 20% rework, Horthy argues, creates an intellectual and emotional burden for both reviewer and author—especially when the reviewer has to untangle generated code whose intent and shape were never settled.

This is where he sees genuine leverage. Models can help teams gather context, draft product and design documents, explore alternatives, produce architecture artifacts, plan slices, and accelerate implementation. Upfront alignment can itself be faster because AI makes it easier to assemble relevant information. Coding is faster because the model performs it. Review is faster because the resulting change is legible and constrained.

The team still reads the code. It also still owns the codebase, rather than outsourcing that ownership to a test suite and a queue of autonomous agents.