Teams Can Reserve Human Review for Risky Pull Requests

Nick Nisi sees code review as a growing constraint on software delivery. Code is being written faster and more pull requests are being opened, while CI, linting, tests, and automated review tools already check much of the code. Yet a pull request can still wait for a human approval after those checks are done. Nisi showed a review screen where tests had passed and automated reviewers had completed their checks, but the change had been waiting on a person for two days and ten hours.

His question is not whether humans can still review code. It is why a human needs to inspect every change when the code itself is increasingly produced by agents and the functionality is what matters. Nisi says engineers are becoming managers of the coding process. He is encouraged by automated reviewers such as Greptile and CodeRabbit: when Greptile gives a pull request a five-out-of-five rating, he feels confident it is unlikely to break production.

That changes the purpose of review. Nisi wants to know whether a change does what he expects, rather than scrutinize its implementation for its own sake. If review remains, he argues, its more important job may be to give people context: what is changing, why it matters, and what the change means to them. He illustrated that distinction with a token-related bug fix: a reviewer might need to know that refresh tokens are now unique and how to test the change, rather than hunt for bugs an automated reviewer could catch.

Nisi does not claim that humans have nothing left to do. Later, he draws a sharper line: tools can identify what changed, but they do not know why the functionality is changing. That “why” remains essential to understanding the change and deciding whether it is appropriate.

Nisi built a code-review demonstration around a simulated fruit fly brain. He began with a brief historical turn: the software term “bug” is associated with a moth found in a relay in the Harvard Mark II logbook in 1947. His pitch for 2026 was to use a bug to find bugs.

The connectome Nisi used was presented as a wiring diagram derived from scans of a male fruit fly’s brain and nerve cord. The presentation described a map containing 176,422 neurons, 6.3 million connections, and more than 125 million synapses. Nisi said the simulation runs the neurons as a model; in the presentation, 450 milliseconds of fly time took about two seconds on his laptop.

Nisi said he left the connectome’s main neural structure untouched. A subset of 31,493 connections in the fly’s memory center could change in response to test results; he said the rest of the connections were not trained. He described the setup as a frozen neural network with a limited memory system that learns from examples.

The fly cannot read source code, so Nisi gave it a “nose.” A separate AI model asks 30 yes-or-no questions about a change. Those answers become a representation of the code’s “smell.” Similar descriptions activate similar cells in the fly’s memory: the presentation described this as roughly five percent of 4,004 memory cells lighting up.

Training supplies the association between a smell and a result. Nisi said he used mutation testing to generate changes against the codebase, then used the resulting diffs as examples for the fly. The AI model converted each diff into answers, and the test outcome supplied the teaching signal. If a test reproduced a bug, the fly got a bitter taste. If the tests still passed, it got sugar. Nisi described the signal as changing how the relevant connections behave: bitter weakens the tendency to approach, while sugar weakens the tendency to avoid.

At review time, the fly responds to the smell. If the smell alone prompts it to extend its tongue, that counts as “smells good to me” and an approval. If it pulls back, that signals a change request. When the response is too uncertain, it abstains.

Nisi said he tested whether the fly’s learned memories mattered by removing them. In his demonstration, that made its behavior fail: the same inputs could produce inappropriate tongue extensions or retractions. He took this as evidence that the memory component was affecting the fly’s responses, rather than the AI model simply supplying a verdict for the fly to display.

Nisi reported testing the system on 301 changes it had never seen. The chart described the task as “telling bugs from toss-ups” and reported 88.3% for the fly, compared with 50% for a coin flip and 84.8% for the best single clue. A spreadsheet reached 94.4%. The fly caught 193 of 234 bugs, approved 63 of 67 harmless changes, and said “I don’t know” 19 times.

The live audience game gave the figures some texture. Nisi drew three changes from the codebase and asked the room whether to approve each one. The audience rejected the first change, but the fly approved it; tests showed that it was a bug. The room and fly both approved the second, which was harmless. For the third, the audience rejected it and the fly also requested changes; it was a bug. The room went three for three, while the fly went two for three.

Nisi also reported results on two harder examples: the fly requested changes 96% of the time on a straightforward mistake and 98% of the time on a plausible mistake. The fly could not explain what was wrong or how to fix it; it could only signal that something “smells funny.” Nisi described a live attempt to submit a real pull request to the system: it returned “Can’t tell.” His invitation to try the tool came with a qualification: people probably should not actually rely on a fruit fly to test their code.

The narrower claim is that the system responded to features such as a changed default, a removed conditional, or deleted logic. Nisi said automated reviewers such as Greptile and CodeRabbit can do that better. But neither the fly nor those tools, in his framing, understands the reasons behind a product or behavior change. A classifier can help sort changes; it cannot replace the team’s understanding of why a particular change should happen.

Nisi’s proposed workflow is to remove humans from routine reviews and escalate the changes that warrant their attention. The first layer is a team-maintained review skill. For example, if a pull request is tagged to WorkOS’s developer-experience team through CODEOWNERS, that team could maintain a script or skill that captures the minimum checks its members would apply.

Next comes a classifier, which could be a larger or smaller model. It would assess whether the pull request is too complex, too risky, or not doing what it claims. Changes that pass could go to an automated reviewer such as Greptile or CodeRabbit. If a check fails—or if the change is too complex—the system escalates it to a person. If the checks pass, the pull request could merge without requiring another person’s routine approval.

That process would need to be inspectable and revisable. If something broke, the team could audit what happened and adjust its review skill or automated checks to catch that class of problem next time. Nisi also suggested using AI to fix code when a problem is found. A human who receives an escalation should get a useful account of the change: its business logic, the reason for it, and alternatives that were considered and rejected. Teams would manage their review skills according to their responsibilities; a developer-experience team and a platform team will look for different risks.

The review also needs evidence about behavior, not just an account of the code. Nisi suggested that agents provide screenshots or videos, test results, command output, and smoke tests that a reviewer can run. They could also give people straightforward ways to reproduce behavior manually. As trust in those checks grows, he expects less manual testing to be necessary.

His example of a developer-experience review skill combined escalation rules with proof requirements. The team might escalate a change that alters the public API, touches sessions, tokens, cookies, or redirects, hard-codes generated SDK code rather than changing the OpenAPI specification, adds a runtime dependency, or is too large to explain in three sentences. It could also require CI to pass across supported framework versions, a video showing the example app signing in and out, documentation and code samples updated in the same pull request, and a changelog line written in language developers would use.

In this model, teams decide what counts as risky and what evidence is enough. Automation handles repeatable checks and assembles proof; humans focus on the changes whose rationale or risk calls for their judgment.