Laurie Voss
Laurie Voss is a software developer and technology leader, co-founder of npm, and Head of Developer Relations at Arize AI. He speaks about developer tools and the rise of the product engineer.
Coding Agents Shift Engineering’s Bottleneck From Code to Product Judgment
Laurie Voss, head of developer relations at Arize AI and co-founder of npm, argues that coding agents have made code cheap to produce without making software delivery equally fast. As agents generate more code, the bottleneck shifts to review, operations and security; the work least suited to automation, he says, is discovering what people need and defining what a good solution looks like. Voss calls the engineer who does that work a product engineer, and warns that the shift is already narrowing the traditional entry-level path into the profession.
AI Agent Evals Turn Confident Errors Into Measurable Failures
Laurie Voss, head of developer relations at Arize AI, argues that teams cannot tell whether an AI agent is improving by checking a few outputs that look plausible. In her workshop, she builds an evaluation loop around a financial analyst agent that combined analyst percentages from separate search results into a supposed consensus, producing figures that totaled 116%. Tracing exposed how the agent reached that answer; evaluations tested its outputs. Voss’s approach also checks those evaluations against human judgment before using them to guide changes and CI release checks.
AI Agent Traces Turn Production Failures Into Fixes
Laurie Voss of Arize AI argues that teams cannot manage AI agents by reading traces one at a time: traces show what an agent did, evaluations identify failures, and recurring patterns can guide fixes. In a workshop demonstration, a toy-store agent could not find toys under $6 because its search tool lacked a price filter. Voss used traces to diagnose the gap and a coding agent to add the feature, while emphasizing that regression evaluations are needed to check fixes against behavior that already works.
A Financial Agent’s 13 Rejected Reports Exposed a Judge Without Research Context
Laurie Voss of Arize AI argues that reliable AI agents require more than trying a few prompts and deciding the results look good: teams need to inspect traces of what agents actually did and define success before automating evaluation. A financial-analysis agent in her workshop produces reports that expose the limits of a judge grading from its own knowledge: it rejects all 13, while a faithfulness check given the agent’s research context finds six faithful and seven unfaithful. Voss uses focused evaluators, human comparisons and repeated tests on saved failures to guide changes before and after deployment.
AI Agents Write 741% More Code, but Review Limits What Ships
AI agents have sharply increased code output without a comparable rise in software shipped: developers using autonomous agents wrote 741% more code, but shipped only 30% more software. Laurie Voss, Arize AI’s head of developer relations and npm co-founder, argues that human review is now the bottleneck—and that asking people to review more code is not a scalable fix. The answer, she says, is to build automated review systems that test more than whether code passes, while keeping human judgment and production monitoring where automated checks fall short.
Longer Skills Files Shift Agent Design From Compression to Verification
Laurie Voss, head of developer relations at Arize AI, argues that frontier models’ capacity to retain simple prompt instructions has risen from roughly 200–300 constraints to thousands in about a year. Re-running and extending the IFScale benchmark, he found that the old compression problem has largely receded, but reliable compliance has not: models now fail through omissions, safety refusals, exhausted reasoning budgets or polished-looking partial answers. His practical conclusion is that teams should verify requirements in outputs rather than assume a fluent response followed the prompt.
Choosing The Right Eval Matters More Than Tuning The Judge
Laurie Voss of Arize argues that agentic applications need the same engineering discipline as other production software: instrumentation, inspectable traces, targeted evals, and controlled experiments, not a handful of prompts that “look right.” In a hands-on workshop using a financial analysis agent, Voss shows how teams should read traces before writing evals, classify failures by root cause, and combine deterministic checks, LLM judges, custom rubrics, and human-labeled meta-evaluation. His central warning is that the choice of eval can dominate the result: the same agent scored 0 out of 13 on a correctness eval and 13 out of 13 on a faithfulness eval because the first judge was asking the wrong question.