Coding Agents Need Executable Proof Before They Can Call Work Done

An agent can make a change that looks successful, say it is done, and still leave the work broken. Nick Nisi and Zack Proser’s central proposal is to make completion depend on a check the agent can run—not on the agent’s description of its own work. The check should exercise the application, inspect what happened, retain evidence, and give the agent a concrete next step when something fails.

The distinction matters because code review alone does not scale cleanly with agent output. Nisi says agents produce code faster than people can review it, while reviewers may become less familiar with code they did not write. A passing build or test suite is useful, but it does not establish that a feature behaves as intended. In their example, a waitlist form displayed a success message after submission. Reloading the page revealed that the email had never been saved. The interface had briefly looked right; the user’s request had not persisted.

The workshop’s opening slide contrasted a “very convincing completion report” with a worker report containing specific checks: the form renders, submission works, and a success message appears. That is more informative than “Looks done to me,” but it can still omit an essential condition. If the request must survive a reload, the check needs to include one. The demonstration made the omission concrete: the page briefly showed the submitted email, then returned to an empty state after refresh.

That gap is easy to miss when people are juggling multiple tasks. Proser describes the subtle failures that appear when an agent is one of many active workstreams: it may say the code looks good, the build succeeds, and the tests pass, while the feature is still wrong. The problem is not only that models make mistakes. A person managing several agents may not have the attention to retrace every user journey and catch each one.

The presenters connect verification to a more consequential concern: trust is a prerequisite for letting agents work with less supervision. Their opening slide described an April 2026 incident in which, according to the slide, an agent deleted production data through a production API while undertaking a routine staging task. The slide said the event took nine seconds, characterized it as “one destructive API call,” and said the instructions did not enforce the boundary. It also said the data was later recovered. Nisi’s point was that instructions are not an adequate boundary; the slide added that recovery is a system capability, too. Proser said verification is not only about preventing errors but, in his view, is foundational to scaling agent work.

Nick Nisi and Zack Proser describe trust as something teams build through evidence, not assume in advance. Nisi calls it a gradient rather than a switch from no trust to complete trust. Proser argues that as model capabilities advance, teams need to give agents a working definition of “good” and feedback that identifies when they have missed it. The goal is not to remove human judgment from every decision. It is to make routine checks and repairs less dependent on a person watching each step.

A slide about Nisi’s tool Riker stated the distinction in its own words: “Status comes from records, not model prose.” The slide presented that as the tool’s principle, not as a spoken transcript quotation. An agent’s statement that work is ready is a claim; a check’s output, tied to the code it evaluated, is evidence. The system needs to make that evidence inspectable and decide what to do with it.

The workshop’s “Midnight Snack Lab” made the gap tangible. The practice app let a user enter an email and join a fictional waitlist. After submission, the page displayed a success message and the email. On reload, both were gone. The interaction had changed the visible interface without persisting the request.

The example illustrates why a browser check should follow the user’s path far enough to expose side effects. Nisi and Proser asked what a developer would ordinarily inspect before shipping a web feature. The answers included reloading the page, checking local storage, looking at console errors, inspecting network requests and response codes, and reading the page itself. An agent with only a terminal cannot do those things unless it has suitable tools. The task is to expose relevant observations through tools the agent can run.

Proser recommends building a lever rather than relying only on a paragraph of instructions. “Verify the UI in a browser” leaves room for interpretation. Adding details—reload the page, check persistence, inspect the console—helps, but still relies on the agent to remember and execute each instruction consistently. A script or CLI command can make the sequence repeatable, return structured results, and tell the agent what failed. Nisi added that one command backed by actual shell calls gives different agent runs a more uniform check than a prompt alone, even though the agents’ behavior is not deterministic.

For the workshop, the speakers built snack, a small, application-specific remote control for the practice app. It was a template for the pattern, not a general-purpose tool that would work for every application. Its command groups covered several stages of a check:

• pnpm snack up started the local app and left it running. Its output included the app URL and a Chrome DevTools Protocol URL, which let the other commands interact with the browser.
• pnpm snack doctor checked setup details such as Node, Chromium, the app session, and whether the page showed the current source. The presenters emphasized that a check against an unhealthy setup or the wrong build does not establish that the intended change works.
• Commands such as pnpm snack join ada@example.test, pnpm snack reload, and pnpm snack reset drove the waitlist and reset its data. The join command could return what happened in the app, including the status, email, and confirmation text.
• pnpm snack snapshot, pnpm snack storage, pnpm snack console, and pnpm snack network exposed different observations of the running app. The snapshot returned a structured view of the page, including its text and accessible elements; storage showed what was saved; console and network commands exposed errors and requests. pnpm snack screenshot captured the page for visual inspection.
• pnpm snack prove ran a deterministic proof, while pnpm snack judge asked a model to assess a question that was not readily expressed as a simple assertion.

The practical sequence was to leave the app running in one terminal, then use another terminal to check readiness, interact with the app, and inspect the result. The slides encouraged participants to predict what each command would print before running it. For example, after joining the waitlist, they could inspect storage, reload, and take another snapshot. The displayed page might say “waitlisted,” while the storage command returned {}. That mismatch showed that the request had not survived reload. In the demo, a command could report a successful-looking interaction while the follow-up inspection exposed the missing persistence.

The tool’s basic loop was presented as four steps: build the check, give the agent a way to run it, keep the evidence, and decide when to stop. Those steps address different gaps. Without a check, “done” is self-reported. Without a way to run it, the check remains a human-only procedure. Without evidence, a pass or failure is hard to inspect or compare with a later attempt. Without a stopping rule, the agent may keep guessing or declare success without meeting the requirement.

A good proof should use the user’s route through the application: enter an email, submit the form, reload, then check what remains. It should verify side effects as well as visible text, and it should perform the trigger and inspect the resulting state in the same run. The presenters also emphasized checking that the app is ready before relying on a result. Evidence for the wrong build or an unhealthy setup does not establish that the intended code works.

The slide titled “A screenshot of success isn’t proof” made the standard explicit. Use the user’s path, not an internal shortcut. Check side effects and what survives reload. Keep the trigger and end state in one run. Run the doctor first: if the setup or build is wrong, the output cannot establish that the intended code works. A screenshot can help a person understand what happened, but it does not by itself show how that state was reached or whether a user can reach it.

That qualification matters because a screenshot could show a state reached through an internal shortcut that a user cannot take, or conceal a side effect that breaks after reload. The check needs to establish how the state was produced and whether the relevant behavior survives. For a web application, that may mean a real browser and a repeatable user journey; for other systems, it means finding the equivalent observable path.

This approach also changes how the work is specified. Proser cautions against confusing a clear outcome with a prescriptive implementation. In his view, newer models can be creative about how to reach a goal, and instructions about every step may constrain them unnecessarily. The useful boundary is the outcome and how to recognize it: a submitted email must persist and be verifiable after reload. The agent can choose the implementation, but it must be able to demonstrate that it arrived.

The check itself can begin as a familiar end-to-end test. In the slide’s example, a Playwright-based prove function checks that a fresh visitor starts with no waitlist status, fills the email field, submits the form, and checks that the page reports the expected result. The exercise asked attendees to add the reload and then check the status, saved email, confirmation, and storage. Its check() function was designed not to throw, so it could collect results rather than stop at the first mismatch; storage() read local storage over the browser connection. The point was not to make a universal proof function. It was to expose the application-specific user journey as something an agent can run again after a repair.

Nisi and Proser describe the working sequence as “reproduce it, fix it, show me proof.” The check can fail first; that is useful if it reproduces the defect, identifies what did not hold, and gives the agent a basis for repair. After the change, the agent runs the same check again. This resembles a red-green-refactor workflow, but the emphasis is on making verification itself executable and available to the agent.

The failure should say what to do next where possible. One example in the slides reported that “status survives reload” expected waitlisted but observed none; another check found no record in local storage. The report’s next-step hint was to persist the record. It also included a screenshot path and distinguished a failed check from a setup that could not tell. As the slide put it, missing evidence never becomes a pass.

Not every claim about an application is equally easy to verify. Some outcomes can be checked directly: whether an email remains in storage after reload, whether an API returned an expected response code, or whether a pull request passed its required checks. For those, a deterministic check can state what it observed and compare that with an expected value. If the status after reload is none when it should be waitlisted, the mismatch is explicit.

Other questions require interpretation. In the waitlist example, a test might confirm that the status reads “waitlisted” after reload. But the screenshot could say, “You’re in! Your seat is confirmed,” even though the application is only a waitlist. The persistence test passes; the user-facing copy misrepresents the outcome. The two observations conflict. The presenters used this example to distinguish checking a directly observable state from judging whether a claim is honest. A rigid assertion about one exact string may not be the best way to judge every possible version of the copy.

The presenters introduced a “judge”: a fast model asked a narrow question about evidence, such as whether the displayed confirmation overpromises. Proser used the System 1 and System 2 distinction as an analogy. System 1 suggests a quick judgment; System 2 suggests slower, more deliberate reasoning. Applied to the workshop’s example, a judge could consider whether the visible copy matches the waitlist’s actual promise. The speakers presented this as a way to assess claims that require interpretation, alongside direct checks of application state. A useful division, based on their examples, is to assert what can be observed directly and use a judge for a narrow interpretive question; the judge does not replace the underlying evidence.

Nisi described using model-generated structured output in his own work, with unit tests around the output to catch edge cases and confirm that it represents the intended interpretation. Those tests can serve as a check against drift. His example was not a claim that every judgment can be made deterministic. It was a way to put boundaries around model interpretation where the team knows what it expects.

The presenters also used a “Saboteur vs. Detective” exercise to make verification practical. The app received a randomly selected bug, and participants had to find it using snack commands alone. Possible faults included memory-only storage, a wrong email, lost confirmation, overpromising copy, a console error, or no bug at all. The intended scoring rewarded naming a bug with output, penalized an unsupported “it works” when the app was broken, and treated “nothing to find” as a legitimate outcome. Confidence alone was not the point; the detective had to show what the tools revealed.

The exercise exposed another requirement: proof must be tied to the exact revision it checks. A previous run may have passed, but its receipt cannot establish that today’s code works. The snack doctor output could label a receipt stale when its recorded revision differed from the current source. Nisi’s slide made the contrast concrete: the old receipt said “PASS” for one revision, while the current source had a different revision. The “perfect receipt for the wrong code” was stale evidence, not a reason to trust the new version.

A receipt makes this relationship visible. In the workshop’s example, a receipt recorded the revision, time, observed status after reload, whether a record was in storage, page errors, the checks passed, and a hint to persist the record before trying again. Those were fields in the demonstration, not a universal required schema. The broader purpose is to bind the claim to the revision tested and preserve the observations and decision: what the app did, which checks passed or failed, and what the result suggests doing next.

Each attempt should add evidence for that attempt rather than turn an earlier result into proof of later code. That leaves a history of what was tried, what happened, and which revision was evaluated. In the workshop’s sample failure, the report showed a FAIL verdict, the revision, a check named “status survives reload” with expected waitlisted and actual none, a second check showing no local-storage record, a next-step hint, and a screenshot path. It also distinguished a failed check from an inability to tell.

The stopping rules shown in the workshop make the same distinction between outcomes. A pass on the current revision, an honest screenshot, and no remaining mismatch may be enough to mark the task done. If the doctor is red or the proof cannot tell, the slide says to fix the setup rather than guess. If the judge is unsure, a human should read the screenshot. The slide also gives an example escalation threshold: if the same check still fails after two fixes, hand a human the receipt. That is a proposed rule in the workshop’s skill, not a universal threshold for every team or risk level.

Clear error messages matter because agents can act on them. If a failure describes the observed state, the expected state, and what to try next, an agent has information to use in a repair and can rerun the check. If it reports only “failed,” a person or agent must investigate before any repair can begin. Nisi argues that useful, specific errors are especially valuable when an agent is expected to continue without constant supervision.

Proser argues that receipts can also support later evaluation. Runs can show how a user journey behaved, where it failed, and what an agent changed to fix it. He described the possibility of aggregating receipts to find repeated failures and build better evaluations. Saving logs alone does not improve a system: the records need to be reviewed, labeled, and turned into cases with expected outcomes. Changes to prompts, tools, or policies can then be tested against the same cases, including untouched examples, before a new baseline is adopted.

Nisi makes the related point that tests, evals, smoke checks, and other verifiers should accumulate into a specification of what the application does. New work can then be checked against earlier expectations, reducing the chance that a feature fix creates a regression elsewhere. Proser described curated run data as a potential advantage for teams using self-evolving systems, while the presentation explicitly treated fine-tuning as optional. Evaluation sets and grading rules need to be maintained; they do not emerge automatically from saved logs.

The speakers’ own tools illustrate two ways of making verification authoritative. Zack Proser described CTX, an experimental web service that began as a shared memory layer across agent harnesses and developed into a place to centralize verification. An agent can report what it completed and ask for a check. A verifier can query a system such as GitHub to confirm that a pull request exists and that its merge checks pass. CTX marks a workflow node satisfied only when the verifier accepts the evidence; otherwise, it returns outstanding failures to the agent.

The CTX diagram separated agent-session facts and artifacts from the verification gate. A bare “done” report could be retained as part of the record, but it was not evidence that satisfied a workflow node. In the example Proser described, a verifier checked GitHub for the relevant pull request and its merge status. The app then showed the repository and pull request it had checked. He suggested that teams might build reusable verifiers in a shared library, skills, or tooling, but described CTX itself as an experiment.

Nick Nisi’s Riker is aimed more directly at supervising engineering work. He described handing it a goal and a way to verify that goal, then letting it manage tasks such as responding to review comments, rerunning checks, and confirming that changes still satisfy the desired behavior. In the presentation, Riker’s status did not come from the worker agent saying “ready.” It came from Riker running the job’s check and keeping the command, exit code, commit, and output hash. If no check ran, the result stayed unverified.

One slide showed a worker handing back a claim that a deployment command was ready, while also saying it had not run the full check. Riker’s status panel said “Nothing counts yet.” The later example showed a job check set before the worker wrote code, then run against the resulting tree. The proof included the exact process and command, exit code, and revision information. When signing rewrote commits but left the tree the same, the slide explained why the proof still held for that tree. The specific mechanics were part of the Riker example; the general distinction was that a worker’s words did not set the status.

For both tools, an agent’s statement is a claim, not the evidence that determines status. That separation matters when a worker reports that a deployment is ready but admits it did not run the full check command. Riker can preserve the report without treating it as proof. The verifier’s record, tied to the relevant revision, determines whether the task is done.

Nisi says his emphasis is on results rather than code in isolation. Code that looks good may fail to deliver the intended behavior; code that is not ideal may still produce the desired result. He wants tangible evidence—tests, smoke checks, screenshots, or recordings—of what the system does. Proser’s CTX design similarly puts verification in a reusable service so different agents can work against shared criteria. Both projects are experiments, not claims that one tool or architecture has become a settled standard.

The speakers do not present a universal verifier. They advise teams to adapt checks to their applications and to the outcomes they care about. A web app might need browser automation, storage inspection, and performance traces. A native client could require saving a draft, quitting, relaunching, and checking the restored state. An API or background job might need to send the same event twice and verify the resulting database or job records. The workshop’s examples were blueprints for building wrappers around the harnesses and tools a team already uses.

For web teams, Proser calls a custom verifier CLI one of the highest-leverage starting points. The CLI should expose the kinds of evidence a developer would inspect: page state, network activity, console errors, and performance. If an interaction feels slow, for instance, the agent can record the real action, inspect expensive call stacks, make a repair, then record again under the same conditions and check behavior against a time budget. The goal is to make otherwise invisible runtime behavior inspectable, rather than ask an agent to guess at the cause of a slowdown.

How to ensure agents use checks depends on the team’s workflow. In the Q&A, Nisi said he would be cautious about relying on skills alone, since it is up to the model whether to follow them. CI can enforce checks; teams can also use hooks or shared agent instructions that say when to run them. He said a Git hook was one possibility, though not the first he would reach for, and mentioned hooks for local agents as another option. He suggested a shared AGENTS.md as a straightforward way to tell agents what to verify after significant changes or before opening a pull request.

Proser described CI as one feature of a broader system, not its only purpose: the immediate value is faster local feedback while an agent is still working. Nisi’s example was to give the agent a task in terms of the desired end state—an email should be added to the waitlist, persist, and be verifiable—then let it use the check while developing. The command can also run in CI, but the local loop can catch a failure before the work is handed off.

For Nisi, that faster feedback is what makes parallel work practical. He described working with several agents at once and wanting to return to their tasks minutes or hours later without restarting the cycle of checking, correcting, and re-prompting each one. Proser similarly wants to finish a day with workstreams that are correctly done, rather than spend the day nudging each agent turn by turn. Verification does not eliminate human review; it aims to bring work to a reviewable state with fewer rounds and better evidence.

Nisi compared that confidence with the way he uses CI and automated code review. He said a high score from Greptile alongside passing CI made him feel confident, but also pointed out that a large amount of code could pass without delivering the intended result. Verification of the outcome is meant to keep the agent on track as well as reduce the number of turns required to get work into a state a person can review.

The same logic could extend beyond merging code, though Proser presented production monitoring as an emerging possibility, not a settled practice. Depending on the stack, a team might verify a preview environment before deployment and monitor the rollout afterward. He described a pattern in which an agent specifically watches production metrics and feature flags and identifies when a deployment is bad and should be reverted. The presenters acknowledged the unease involved, but argued that there is a path toward broader autonomy if the system has checks, retained evidence, and a way to respond when reality diverges from the expected result.