AI Engineer
Talks, workshops, events, and training for AI engineers.
AI Evaluations Need Human Calibration Before They Can Gate Deployments
Tejas Kumar of IBM argues that AI evaluations should be treated as calibrated reliability checks, not as proof that a system works. In a customer-support example, a test passes an answer that wrongly approves a return because it contains the word “cannot”; an LLM judge can also mislead unless its verdicts are checked against human-labeled cases and the relevant policy. Kumar’s approach is to examine disagreements, refine the judge and dataset, and use measured agreement as a deployment gate.
Coding Agents Need Testable Definitions of Done
Nick Nisi and Zack Proser of WorkOS argue that engineers can delegate more work to coding agents only when they define completion in terms the agents must verify. Their workshop turns a failed test loop—an agent touched a file meant to prove tests had run, without running them—into a broader operating rule: use measurable goals, automated checks and human review to keep parallel and recurring work in bounds.
AI Agents Write 741% More Code, but Review Limits What Ships
AI agents have sharply increased code output without a comparable rise in software shipped: developers using autonomous agents wrote 741% more code, but shipped only 30% more software. Laurie Voss, Arize AI’s head of developer relations and npm co-founder, argues that human review is now the bottleneck—and that asking people to review more code is not a scalable fix. The answer, she says, is to build automated review systems that test more than whether code passes, while keeping human judgment and production monitoring where automated checks fall short.
The Best Agent Workflows Start With Context, Guardrails, and Human Judgment
Charlie Holtz, co-founder and CEO of Conductor, argues that teams using coding agents should build around human judgment rather than treat software development as a factory line. His principles include staying close to new tools without chasing workflows that will soon become standard, protecting code and agent instructions with strict review, and giving agents access to organizational knowledge and persistent workspaces. The aim, he says, is for people to conduct teams of colleagues and agents, not simply manage automated production.
Managing Coding Agents at Scale Requires Better Visibility, Autonomy, and Review
Ido Salomon, creator of AgentCraft, argues that the obstacle to using coding agents at scale is the human effort required to steer, direct and review them. His strategy-game-style orchestrator represents agents as units on a map, with the file system as terrain, and combines visibility tools with autonomous workflows and shared workspaces. Salomon says the detailed interface may suit power users, but not everyone; an early, simpler mode is intended to make agent tools easier to approach.
Agents Multiply Pull Requests Faster Than Teams Can Review Them
Pull requests are becoming a bottleneck as agents increase the flow of changes faster than teams can review them. In a talk at AI Engineer Paris 2026, Matt Pocock argues that the answer is not simply to generate code faster, but to make changes more reviewable: run deterministic checks, then agent review, before human judgment. Human reviewers can focus on risk and use recurring findings to improve the process behind future PRs.
Action-Based Search Could Make Web Video Usable for Robot Training
Rafael Levi of Bright Data argues that robotics’ training-data bottleneck is not simply a shortage of video, but the difficulty of finding useful footage in the vast amount already online. Staged demonstrations can distort how people naturally perform tasks, while broad web-video downloads often discard most of what they collect: Levi cites rates of 96% for NVIDIA’s Cosmos pipeline and 74% for Stable Video Diffusion. His proposed alternative is to search for actions first and collect only matching clips, while retaining the need for robot-specific data and further processing.
Task-Specific Detectors Can Beat Vision-Language Models at Runtime
Merve Noyan, a computer-vision and multimodal systems researcher at Hugging Face, argues that developers should stop deploying vision-language models as the default for vision tasks. They are too slow for many real-time uses, she says, and a task-specific detector can be faster and more accurate; licensing can also make popular model choices unsuitable. Her VibeVision toolkit instead gives coding agents task-specific models and a workflow that turns unlabeled images into a trained detector, with human checks on key decisions.
FlashNorm Folds RMSNorm Into Projection Weights to Cut Decode Time
Filip Makraduli argues that RMSNorm’s inference cost comes less from arithmetic than from repeated GPU launches, memory movement and dependencies that hold up projection matmuls. His FlashNorm approach exactly rewrites the norm–projection sequence by folding learned gains into weights, deferring the RMS scalar division until after the matmul, and in some architectures removing a redundant normalization. Weight folding can speed existing checkpoints now, he says, while the larger benefit from deferred normalization depends on fused kernels and correct CUDA-stream synchronization.
Agent-Ready Interfaces Can Override a Decade of Brand Loyalty
Liad Yosef, co-creator and maintainer of the MCP Apps specification, argues that online services will increasingly compete on how readily agents can discover, authenticate with, and operate them—not just on the quality of their human-facing dashboards. He says companies should test agents completing real tasks rather than rely on published standards such as `llms.txt`, which Ora found agents often ignore. In Yosef’s model, services become “nearly headless”: agents handle routine operations directly, while providers return an interactive interface in chat when human judgment is required.
Full-Duplex Voice Agents Must Separate Conversation From Reasoning
Neil Zeghidour, co-founder and CEO of Gradium, argues that most real-time voice agents remain “walkie-talkies”: however quickly they respond, they can only listen or speak, not handle the overlapping signals that make human conversation work. Full-duplex speech models can represent simultaneous talk and backchanneling, he says, but they still sacrifice reasoning and tool-use capability relative to text-based agents. His proposed answer is a split architecture in which a lightweight speech model manages the live conversation while a separate text model handles harder reasoning and actions.
Voice Agents Should Act Through Interfaces, Not Just Speak
OpenAI developer-experience lead Charlie Guo argues that voice agents should not be designed to answer every spoken request with speech. The underused model, he says, is speech to action: a user states an intent, the system calls tools and changes the interface or completes a workflow, using audio only where conversation, hands-free operation or escalation warrants it. For developers, Guo’s practical case is that many of those tools already exist as application endpoints, handlers and workflows.
Managed Sandboxes Replace Agent Tool Registries With Skills and Filesystems
Google DeepMind’s Philipp Schmid argues that agent builders should stop encoding anticipated workflows in custom functions and instead give capable models a managed sandbox, general-purpose tools, clear instructions and tightly defined access. In his three versions of a GitHub pull-request reviewer, Python loops, tool schemas and registered functions give way to an `AGENTS.md` file, skill files and a remote environment that handles execution, state and credential injection. The work that remains with teams, he says, is setting behavior, permissions and evaluation—not maintaining orchestration that improving models can replace.
Adversarial Agents Turn Moral Principles Into Testable Case Law
Brendan Rappazzo, a machine-learning researcher at Morgan Stanley speaking about an independent open-source project, argues that people can often judge a concrete moral case but cannot anticipate every case their stated principles must cover. His tool, Loophole, translates plain-language values into a formal code, then uses adversarial agents to find conduct that is immoral but permitted or moral but prohibited. A judge agent patches drafting errors where possible and sends unresolved value conflicts back to the user for a decision.
Longer Skills Files Shift Agent Design From Compression to Verification
Laurie Voss, head of developer relations at Arize AI, argues that frontier models’ capacity to retain simple prompt instructions has risen from roughly 200–300 constraints to thousands in about a year. Re-running and extending the IFScale benchmark, he found that the old compression problem has largely receded, but reliable compliance has not: models now fail through omissions, safety refusals, exhausted reasoning budgets or polished-looking partial answers. His practical conclusion is that teams should verify requirements in outputs rather than assume a fluent response followed the prompt.
Agent Autonomy Makes Organizational Context an Engineering Requirement
Brandon Waselnuk of Unblocked argues that the main constraint on AI coding agents is not model intelligence but missing organizational context—the accumulated knowledge of conventions, decisions and incidents that human teammates acquire over time. In Unblocked’s same-prompt test, adding a context engine cut token use from 20.9 million to 10.8 million and reduced runtime from two hours and 33 minutes to 25 minutes. He contends that curated documentation and MCP access alone do not solve the problem, because agents need to weigh conflicting, current and permissioned information rather than merely retrieve it.
Collective Agents Raise the 11-Dimensional Kissing Number Bound to 604
James Zou of Stanford and Together AI argues that agents should be given environments with shared information, deterministic verification and live incentives rather than prescribed workflows. His EinsteinArena, which admits only AI agents, let agents refine one another’s work to raise the 11-dimensional kissing-number lower bound from 593 to 604 and has also produced production GPU kernels with speedups above 2x, he says. Zou applies the same premise to DSGym, where execution-verified data-science tasks are meant to prevent benchmark shortcuts and generate training traces.
Agent Evals Must Raise the Floor, Not Chase the Ceiling
Ben Hylak of Raindrop argues that agent teams should focus less on headline capabilities than on raising the “floor”: the product-specific failures that break trust when agents can act through tools, permissions and external systems. Because agents can produce effectively unlimited issues, he says teams need to know when a failure began and what share of users it affects, then evaluate the full harness in code rather than rely on brittle chatbot-era test suites.
Automation and Weaker Scaling Returns Could Widen Frontier AI Access
Sara Hooker of Adaption argues that frontier AI has been constrained by tacit training expertise and the concentration of large-scale compute, leaving fewer than 5,000 people able to train models at the frontier. She contends that automated, data-aware training systems such as Adaption’s AutoScientist, together with diminishing returns from ever-larger pretraining runs, could shift advantage toward adaptation, domain data and experimentation. Hooker does not argue that compute or large models no longer matter, but that the most productive uses of compute are becoming more distributed than giant colocated pretraining runs.
Long Tool-Use Trajectories Expose Self-Distillation’s Stability Limits
Ronak Malde of Trajectory argues that on-policy self-distillation can make production agent traces usable as dense training signal: a model learns from its own trajectories by matching a version of itself given privileged hints. He says the method avoids the parallel rollouts and trajectory-level rewards of GRPO, but breaks down on long, tool-using tasks when the teacher repeatedly corrects a student that has drifted off course, producing what he calls the “but wait” problem. Malde’s proposed remedies—step-level divergence weighting and residual guidance—are meant to preserve useful correction without teaching the model to hedge or exploit answer-revealing hints.
Context, Permissions, and Network Overhead Define Agent Performance
OpenAI’s Dominik Kundel argues that an agent’s practical capabilities depend as much on its harness as on its model: the surrounding system determines how context is assembled, which tools are available, what actions require review, and how work persists across turns. In Codex, that means deferring tool definitions to protect context, matching file and shell interfaces to model training, and using constrained review agents for sandbox escalations. As inference accelerates, Kundel says, the harness must also reduce network overhead through stateful WebSocket transport and manage long-running work through explicit goals and context compaction.
MCP Apps Standardizes Host-Controlled UI Across AI Assistants
Liad Yosef and Ido Salomon argue that MCP Apps gives AI assistants a way to render service-specific interactive interfaces rather than reducing every result to text. Built from the MCP-UI project and being developed as an MCP extension, the approach lets a server return an app resource that a compatible host renders in a sandbox, while clicks and other actions return through the host for further orchestration. They present it as a portability and distribution model, though state management and interoperability remain active work.
Automated AI Research Shows Early Gains in Models, Training, and Kernels
Richard Socher, CEO of Recursive AI, argues that science is constrained by the number of people able to work on increasingly specialized problems, and that agent systems could relieve that bottleneck by running the cycle of hypothesis, experiment, validation and revision at scale. His proposed “Eureka machine” would begin by automating AI research, using search, compute and software tools rebuilt for agents rather than human workflows. The benchmark gains he presents in model training, architecture search and CUDA kernels are early proof points, he says—not yet recursive self-improvement or autonomous science.
RAM, Not Compute, Is Constraining Edge AI Deployment
Cormac Brick of Google AI Edge argues that RAM—not compute—is becoming the binding constraint on local AI, forcing developers to fit models within the memory budgets of phones, robots and low-cost devices. Models in the one-to-four-billion-parameter range can support prompt-based features on better-equipped hardware, he says, but broader deployment requires 50-million-to-500-million-parameter models fine-tuned for specific tasks. The trade is less generality for faster, cheaper offline capabilities such as voice control and dictation.