AI Engineer
Talks, workshops, events, and training for AI engineers.
Playwright Lets Agents Test Feature Requests Before They Write Code
Microsoft’s Marlene Mhangami argues that AI-generated tests can make a codebase look healthier than it is, because agents often write tests that confirm their own implementation rather than validate the user-visible behavior a feature is meant to deliver. Her prescription is to reverse the common workflow: start from the feature request, have the agent write failing Playwright tests against expected behavior, then generate code to pass them. In a GitHub Copilot demo using the Playwright MCP server, she applies that approach to a toy-store search and filtering feature, with the browser showing the agent exercise the product experience directly.
Vertical AI Teams Need Domain Experts Who Own Quality Loops
Chris Lovejoy of Notius Labs argues that vertical AI companies increasingly fail or succeed on whether they can turn domain judgment into product quality, not simply on access to better models. He proposes three operating models for that expertise: an Oracle who both judges and changes outputs, an Evaluator who defines and measures quality while engineers implement fixes, and an Architect who designs systems that improve from use. His case studies of Granola, Tandem and Anterior show why the right model depends on whether quality is subjective, measurable, or too variable for manual iteration.
Context Graphs Make AI Decision Trails Queryable
Stephen Chin of Neo4j argues that enterprise AI systems need context graphs because retrieval alone can surface relevant facts while missing the relationships that make them usable. In his examples, a graph-augmented system can connect a patient’s emphysema care plan to smoking history or a credit decision to prior rejections, policies, margin trades and fraud signals. Chin’s case is that agents should preserve not only documents and answers, but the decision traces, tool calls, causal chains and outcomes that let humans inspect and reuse prior reasoning.
PFF’s Two-Engineer Agent Team Shipped 10x More Output
PFF CTO Mike Spitz argues that AI agents change the basic operating constraint of an engineering organization: the question is no longer how to make engineers faster, but how to make agents faster. In a three-month case study, he says two agent-heavy engineers shipped far more frequently than a ten-person team on the same codebase, with PFF measuring a 10x output gain per engineer and higher customer satisfaction. The result, in his account, was not the end of engineers but the removal of Scrum-era coordination rituals and a sharper split between agent-executed work and human judgment.
Supabase Says Skills and MCP Close the Agent Context Gap
Pedro Rodrigues of Supabase argues that agents fail on production systems less because they cannot reason than because they lack product-specific judgment. In a test using the same Postgres task, Supabase found that Claude with MCP alone created a view that could bypass row-level security, while MCP plus a Supabase skill added the required `security_invoker = true` flag. Rodrigues’s case is that MCP gives agents tools, but skills supply the rules, workflows, and current documentation paths needed to use those tools safely.
Intercom Doubled Engineering Throughput by Standardizing on Claude Code
Brian Scanlan, a senior principal engineer at Intercom, argues that the company doubled engineering throughput by treating AI coding as an internal platform strategy rather than an individual productivity tool. In his account, Intercom standardized on Claude Code, encoded recurring engineering work into agent-usable skills, connected agents to internal systems under existing controls, and made AI adoption an explicit expectation across R&D. The reported result was a doubling of pull-request throughput, including 17.6% of merged PRs approved by Claude, alongside new bottlenecks in review and CI.
Choosing The Right Eval Matters More Than Tuning The Judge
Laurie Voss of Arize argues that agentic applications need the same engineering discipline as other production software: instrumentation, inspectable traces, targeted evals, and controlled experiments, not a handful of prompts that “look right.” In a hands-on workshop using a financial analysis agent, Voss shows how teams should read traces before writing evals, classify failures by root cause, and combine deterministic checks, LLM judges, custom rubrics, and human-labeled meta-evaluation. His central warning is that the choice of eval can dominate the result: the same agent scored 0 out of 13 on a correctness eval and 13 out of 13 on a faithfulness eval because the first judge was asking the wrong question.
Agent Observability Is Moving From Dashboards to Eval-Driven Optimization
Amy Boyd and Nitya Narasimhan of Microsoft argue that agent observability has to track the widening gap between what an AI agent is meant to do and what it actually does as models, prompts, tools and user behavior change. Their walkthrough of Microsoft Foundry frames observability as a loop of OpenTelemetry tracing, trace-linked evaluations, monitoring, optimization and red teaming. The central demonstration is an observe skill that can generate an evaluation dataset, run batch tests, optimize prompts, compare versions and roll back to the best-performing agent version from a sparse starting point.
An Event-Sourced Agent Harness Separates State Replay From Side Effects
Jonas Templestein of Iterate argues that an agent harness can be reduced to an append-only event stream plus processors: synchronous reducers to derive state, and post-append hooks to perform side effects. His design puts model chunks, tool calls, errors, schedules, subscriptions and even processor deployment into the log, so a restarted agent can replay state without replaying old LLM calls. The larger claim is that agents and third-party services can compose by reading and appending to the same durable stream, with bounded waits and circuit breakers replacing tighter, blocking plugin interfaces.
Agents Can Now Fine-Tune Open Models Through Prompted Workflows
Merve Noyan argues that open models have moved from downloadable artifacts into an operational stack for selection, serving, inspection, training and deployment. In her Hugging Face presentation, she makes the case that access to model weights now matters because developers can quantize, fine-tune and run models locally or at the edge, while Hub benchmarks, inference providers, traces, MCP and Skills let agents act directly on those workflows. Her strongest example is a coding agent that can size hardware, choose infrastructure and launch a fine-tuning job from a prompt.
Continuous Agents Need Stateful Compute, Not Traditional CI/CD
Madison Faulkner and Hugo Santos of Namespace argue that traditional CI/CD is organized around human-paced pull requests, and starts to fail when autonomous agents generate continuous, overlapping streams of code. Their proposed replacement keeps validation inside a stateful agent loop, uses caching and orchestration to avoid cold starts, and moves completed work into a pre-merge layer where humans review intent and outcome rather than every diff. The underlying CI functions remain, but the pull request stops being the system’s basic unit of work.
Persistent Sandboxes Make Agents Remember, Plan, and Reuse Their Work
Nico Albanese, a Vercel engineer working on the AI SDK, argues that agents become more reliable when they are given a persistent sandboxed computer, not just a runtime and tools. In his workshop, he builds that pattern with AI SDK 6, Vercel’s named sandboxes, a bash tool, and a file-backed memory system, showing how an agent can plan in files, preserve context across sessions, and create reusable scripts without a separate memory layer.
Enterprise GenAI Pilots Fail When Feedback Cannot Reach the Model
Alessandro Cappelli, co-founder and chief customer officer of Adaptive ML, argues that enterprise generative AI pilots fail to reach production because companies lack a systematic way to turn defects, user feedback, business metrics and production signals into model improvement. In a talk on Fortune 500 deployments, he says prompting and instruction fine-tuning can produce credible demos, but reinforcement learning is the mechanism needed to train models and agents against enterprise-specific environments, rewards and KPIs. His case is that agents make this feedback loop more urgent, because they consume more tokens, touch live systems and leave less room for error.
Fixed Evaluation Suites Go Stale as Agents Optimize Toward Intent
Vincent Koc of Comet ML argues that AI evaluation is being outpaced by the systems it is meant to measure. In a talk on adaptive evaluation for agents, Koc says static benchmarks and handcrafted test sets are poorly suited to applications that change with prompts, tools, production traces, user behavior and even their own harnesses. His proposed direction is to define the intended end state, use traces and telemetry to surface drift and edge cases, and treat evals as a continuously revised system rather than a one-time benchmark.
Coding Agents Work Best When Products Expose Simple Tools
Matthias Luebken argues that coding agents such as OpenClaw are less mysterious than they appear: they are LLMs calling tools in a loop, made more useful by a runtime, shell, sessions and product hooks. In his Tavon talk, he uses Pi, a minimal coding-agent SDK, to show how that loop can be embedded inside business software, including a sales workflow where RFP emails are routed to customer-specific agent sessions and returned to users as draft replies. His architectural point is that teams should not force agents through opaque systems, but expose data, commands and controls in forms coding agents can use cleanly.
Slack-Native AI Coworkers Turn Memory and Permissions Into Product Risks
Fryderyk Wiatrowski argues that building Viktor as an AI coworker inside Slack is not a matter of scaling a personal assistant to more users. A company-level agent gains value from shared context, shared integrations, and the ability to act where work is discussed, but those same features create harder problems around memory isolation, permissions, fragmented Slack conversations, proactivity, and tone. His case is that an “AI employee” has to be designed less like a chatbot and more like a new hire entering the company’s communication layer.
Apple-Device AI Is Becoming Viable Without Cloud Inference
Prince Canuma presents MLX, Apple’s array framework for Apple Silicon, as a practical foundation for running AI agents locally rather than through cloud services. His case is rooted in accessibility and unreliable connectivity, but extends to product constraints for voice agents, robots and multimodal apps: vision, speech, video generation and long-context inference can increasingly run on Macs, iPhones and iPads without a network call. Canuma does not argue that local models replace every frontier cloud system, but that the boundary has moved far enough to make on-device AI a serious deployment option.
Durable Agents Need Context Logs and Execution Snapshots
Eric Allam of Trigger.dev argues that durable agents need more than the replay-based workflow model used for durable transactions. In his talk, he separates agent durability into two problems: the LLM context, which fits naturally as an append-only log, and the execution environment — files, memory, subprocesses and local state — which he says should be preserved through OS-level snapshot and restore. Allam uses Trigger.dev’s Firecracker work to make the case that long-running agents are becoming session-like workloads, not just replayable transactions.
Head-Tail Truncation and Memory Stabilized Arize’s Trace-Analyzing Agent
Sally-Ann DeLucia argues that agent performance depends on context management as an operating discipline, not on larger prompts or simple compression. Drawing on Arize’s work building Alyx, an agent that analyzes trace data from AI systems including its own, she says naive truncation broke follow-up reasoning and LLM summarization gave the model too much control over what mattered. Arize’s more durable pattern was to preserve the head and tail of context, store the middle for retrieval, test long sessions explicitly, and move heavy workloads into sub-agents.
Production AI Features Need Feedback Loops, Not One-Shot Prompts
Mehedi Hassan, a product engineer at Granola, argues that the hard part of shipping AI features is not getting a model to work once in a demo, but making its behavior reliable and inspectable in production. Using Granola’s meeting-notes app as the case, he says web search, chat, and prompt personalization quickly expose costs, context limits, provider instability, and role-specific user expectations that a single prompt cannot absorb. Granola’s response, in his account, was to build feedback loops: internal tracing, broadly usable debugging tools, and faster ways to test product variants before shipping.
Text-to-Speech Models Are Converging on LLM-Style Architectures
Samuel Humeau of Mistral argues that modern text-to-speech has converged on an architecture that resembles large language modeling: an autoregressive transformer generates compressed audio tokens frame by frame, rather than raw waveform samples. Using Mistral’s open-weight Voxtral TTS model as the example, he says neural audio codecs make that possible by reducing dense speech signals to token-like representations a transformer can handle. The remaining latency frontier, in his account, is not just streaming playable audio early, but letting TTS consume an LLM’s text stream as it is still being written.
Voice AI Still Confuses Natural Speech With Real Conversation
Neil Zeghidour, CEO of Gradium AI and one of the researchers behind the full-duplex voice model Moshi, argues that voice AI’s long-promised “Her” moment is still being confused with better synthetic speech. His case is that cascaded voice agents are useful but structurally too slow and lossy to feel conversational, while speech-to-speech models improve flow but remain limited unless they can listen and speak simultaneously, use tools reliably, understand paralinguistic cues, and run cheaply enough to scale.
ElevenLabs Voice Engine Wraps Existing Chat Agents Without Rebuilding Them
Luke Harries of ElevenLabs argues that the next step for chat agents is not a new orchestration stack but a voice layer around the agents companies have already built. His case for ElevenLabs’ Voice Engine is that teams can keep their existing LLM logic, RAG, tools and business rules, while offloading speech-to-text, text-to-speech, turn-taking and interruption handling to a wrapper. The product is positioned for companies that want voice interfaces across web, phone and meeting channels without rebuilding their chat agents inside a fully managed platform.
Pretraining and Attention Infrastructure Made Vision Transformers Practical
Isaac Robinson of Roboflow argues that transformers overtook convolutional networks in vision not because images stopped needing visual structure, but because that structure moved from hand-built architecture into pretraining, scaling and tooling. In his account, ViT-style models first lacked the inductive biases and efficiency that made CNNs dominant, but self-supervised vision pretraining and attention infrastructure from the LLM world made the simpler architecture practical. Robinson frames the next problem as deployment: turning large foundation backbones into model families that can meet real latency, cost and hardware constraints.