Evals and Benchmarks
Methods for measuring model and system performance, including benchmarks, task-specific evals, red teaming, reliability tests, and quality measurement.
AI’s Value Is Shifting From Model Demos to Distribution and Measurement
Google’s problem at I/O, Jordi Hays argued, was no longer proving that its AI models are impressive, but making Gemini useful rather than redundant across products investors now increasingly view as part of a full-stack AI business. The TBPN discussion extended that framing across the rest of the show: AI’s value, the hosts and guests argued, depends less on model spectacle than on distribution, workflow integration, economics and adoption by institutions. That distinction ran from Google’s risk of crowding users with Gemini entry points to SendCutSend’s physical capacity constraints, Commure’s push to automate healthcare administration, and METR’s effort to turn frontier-model risk into something auditable.
Spotify Uses Semantic IDs to Make LLMs Recommend Catalog Items
Spotify’s Shivam Verma argues that LLM-era personalization requires translating both users and catalog items into forms a model can process alongside language. In his account, Spotify combines long-term user embeddings, Semantic IDs that turn tracks and episodes into token sequences, and soft tokens that project a listener’s profile into an LLM’s embedding space. The aim is a generative recommender that can produce catalog-native recommendations without full fine-tuning, while still relying on traditional ranking layers for production use.
Serval Bets Boring IT Controls Will Unlock Enterprise AI
Serval founder and CEO Jake Stauch argues that enterprise AI will be won less by giving models broad autonomy than by constraining them inside permissions, approvals, audits and workflows that companies can trust. In a conversation hosted by Sequoia’s Pat Grady, Stauch describes Serval as a ServiceNow-like system rebuilt for AI: an admin agent generates workflows from natural language, while a help desk agent can act only through tools IT has explicitly approved. He says that same logic extends to Serval’s operating model, where customer insight and “fewer, better” hiring matter more than model access in a market that may force products to be rebuilt every few months.
GPT Image 2 Wins on Layout While Nano Banana 2 Wins on Speed
ElevenLabs’ side-by-side test of GPT Image 2 and Nano Banana 2 argues that the models are complementary rather than interchangeable. In more than 20 generation and editing prompts, GPT Image 2 was favored for strict prompt adherence, tight composition, source-faithful edits, and text-heavy layouts, while Nano Banana 2 was faster, cheaper at 4K, and stronger in several tasks involving detail retention, realism, and consistency. The practical recommendation is to A/B the same prompt and choose the model whose likely failure mode fits the job.
UK Government Tests an Insurgent Model for In-House AI Delivery
Eoin Mulgrew of the Number 10 data science team argues that the UK state’s AI problem is less a shortage of use cases than a shortage of technical people with the access, mandate, and proximity to build inside government workflows. In a talk on the No. 10 Innovation Fellowship, he presents the model as a deliberate hack around normal civil-service constraints: market-rate pay, outside recruitment, a highly selective technical process, and authority to enter departments and ship tools that remain with the teams using them.
GPT Image 2 Beats Nano Banana 2 on Control, Not Speed
ElevenLabs’ side-by-side test of GPT Image 2 and Nano Banana 2 argues that the models are complementary rather than interchangeable. Across more than 20 generation and editing prompts, the comparison found GPT Image 2 stronger when briefs required tight prompt control, text hierarchy, layout discipline, and source fidelity, while Nano Banana 2 more often won on speed, 4K cost efficiency, fine detail, and polished editorial transformations. The practical recommendation is to route work by failure risk — and A/B test important prompts — rather than pick a single default model.
Long-Running Agents Need Separate Builders, Evaluators, and Disposable Scaffolding
Anthropic’s Ash Prabaker and Andrew Wilson argue that long-running agents are a harness-design problem, not a matter of writing longer prompts. Their case is that agents can run for hours only when building, judging, planning and state management are separated: adversarial evaluators should test live behavior, work should be decomposed into explicit contracts, and durable state should live outside the model’s context. They also warn that this scaffolding is provisional, because each new model release changes which supports are useful and which have become dead weight.
Incident.io Uses Coding Agents to Debug Its AI SRE
Lawrence Jones, founding engineer at Incident.io, argues that complex AI products now require debugging tools built for agents as well as humans. In a talk on Incident.io’s AI SRE system, which runs hundreds of prompts across telemetry and code during production investigations, Jones describes how the team moved from human trace inspection to agent-addressable evals, downloadable file-system traces, and parallel analysis pipelines to find and fix failures that had become too large to debug manually.
Agentic AI Is Turning Model Quality Into a Systems Problem
At AI Engineer Singapore’s second day, speakers from Google DeepMind, Cloudflare, Arize, OpenClaw, Adaption and other teams made a shared engineering case: as AI systems become more agentic, model quality is no longer separable from the systems around the model. Richard Ngo framed the risk as long-horizon, situationally aware agents whose goals cannot be inspected, while practitioners argued that production AI now depends on continuous evaluation, traces, deterministic execution boundaries, routing, memory, fine-tuning and test-time search. The source’s central claim is that useful and safe agentic AI is becoming a systems problem, not just a model-selection problem.
Vertical AI Teams Need Domain Experts Who Own Quality Loops
Chris Lovejoy of Notius Labs argues that vertical AI companies increasingly fail or succeed on whether they can turn domain judgment into product quality, not simply on access to better models. He proposes three operating models for that expertise: an Oracle who both judges and changes outputs, an Evaluator who defines and measures quality while engineers implement fixes, and an Architect who designs systems that improve from use. His case studies of Granola, Tandem and Anterior show why the right model depends on whether quality is subjective, measurable, or too variable for manual iteration.
Self-Driving Startups Shift From Science Risk to OEM Deployment
Wayve chief executive Alex Kendall and Waabi chief executive Raquel Urtasun argue that self-driving has moved from a basic research problem to an execution problem built around end-to-end AI, world models, OEM partnerships and deployment economics. In this This Week in Startups discussion, Kendall makes the case for licensing Wayve’s “intelligence layer” across consumer vehicles and robotaxis, while Urtasun says Waabi’s L4-native Driver-as-a-Service model can scale first through trucking and then robotaxis. Both reject the idea that autonomy is simply solved, but they present the remaining challenge as integration, validation, regulation and commercialization rather than a missing scientific breakthrough.
AI Cyber Models Push Trump Administration Toward Pre-Release Safety Reviews
Kevin Roose and Casey Newton argue that the Trump administration’s shift toward AI safety is being driven by frontier models that can find and chain software vulnerabilities, not by a broad ideological conversion. Drawing on New York Times reporting about a possible executive order for pre-release model review, they describe a policy scramble over Anthropic’s Mythos, chip access to China and which federal agency should judge dangerous models. Nikesh Arora, Palo Alto Networks’ chief executive, says the cyber problem is already operational: attacks that once unfolded over days may soon move in minutes.
Supabase Says Skills and MCP Close the Agent Context Gap
Pedro Rodrigues of Supabase argues that agents fail on production systems less because they cannot reason than because they lack product-specific judgment. In a test using the same Postgres task, Supabase found that Claude with MCP alone created a view that could bypass row-level security, while MCP plus a Supabase skill added the required `security_invoker = true` flag. Rodrigues’s case is that MCP gives agents tools, but skills supply the rules, workflows, and current documentation paths needed to use those tools safely.
Intercom Doubled Engineering Throughput by Standardizing on Claude Code
Brian Scanlan, a senior principal engineer at Intercom, argues that the company doubled engineering throughput by treating AI coding as an internal platform strategy rather than an individual productivity tool. In his account, Intercom standardized on Claude Code, encoded recurring engineering work into agent-usable skills, connected agents to internal systems under existing controls, and made AI adoption an explicit expectation across R&D. The reported result was a doubling of pull-request throughput, including 17.6% of merged PRs approved by Claude, alongside new bottlenecks in review and CI.
AI Is Moving Deeper Into Science, but Validation Remains the Bottleneck
At AI+Science: AI for the Universe, Kyle Cranmer, Carina Hong and Douglas Finkbeiner argued that AI is already embedded in scientific work, but its value depends on where validation happens. Cranmer framed physics applications around prediction and inference, where formal checks, simulator calibration or uncertainty correction determine whether model output can support scientific claims. Hong made the parallel case in mathematics, where Lean-style formal proof gives some AI results a clean score but leaves problem selection and theory-building with experts. Finkbeiner said astronomy’s newer disruption is the desk-level AI collaborator, which can improve research work while increasing the need for verification and scientific judgment.
AI Is Making Scientific Throughput the New National Advantage
Dario Gil, the U.S. Department of Energy’s Under Secretary for Science, used his AI+Science keynote to argue that AI is shifting scientific advantage from access to instruments and computing toward the throughput of integrated discovery systems. He presented DOE’s Genesis initiative as the national-scale architecture for that shift, linking data, AI models, high-performance computing, experimental facilities, and industry partners into closed-loop workflows. Gil’s case was that the test is not more papers, but whether faster scientific cycles can produce measurable gains in productivity, security, and industrial capability.
AI-for-Science Advances Depend on Evaluation, Not Just Generation
In a Stanford AI+Science lightning-talk session introduced by Surya Ganguli, four young researchers made a common case: AI-for-science is useful only when paired with rigorous evaluation. Aishwarya Mandyam, Amar Venugopal, Steven Dillmann and Alda Elfarsdóttir each treated AI systems or outputs as claims to be tested — through uncertainty estimates for clinical policies, causal checks on generated text, executable benchmarks for scientific agents, and empirical links between corporate climate language and later emissions.
Abridge Bets Clinical Conversations Can Become Healthcare’s Intelligence Layer
Abridge executives Janie Lee and Chaitanya “Chai” Asawa argue that the patient-clinician conversation is becoming healthcare’s core intelligence layer, not merely an input for automated notes. In a discussion with Redpoint’s Jacob Effron, they describe Abridge’s move from ambient documentation into clinical decision support, prior authorization and other workflows that depend on EHR data, payer rules, medical literature and local guidelines. Their case is that healthcare AI will be judged less by chatbot fluency than by whether it can deliver accurate, low-latency, privacy-preserving support inside clinical workflows without adding to clinicians’ alert burden.
Choosing The Right Eval Matters More Than Tuning The Judge
Laurie Voss of Arize argues that agentic applications need the same engineering discipline as other production software: instrumentation, inspectable traces, targeted evals, and controlled experiments, not a handful of prompts that “look right.” In a hands-on workshop using a financial analysis agent, Voss shows how teams should read traces before writing evals, classify failures by root cause, and combine deterministic checks, LLM judges, custom rubrics, and human-labeled meta-evaluation. His central warning is that the choice of eval can dominate the result: the same agent scored 0 out of 13 on a correctness eval and 13 out of 13 on a faithfulness eval because the first judge was asking the wrong question.
Agent Observability Is Moving From Dashboards to Eval-Driven Optimization
Amy Boyd and Nitya Narasimhan of Microsoft argue that agent observability has to track the widening gap between what an AI agent is meant to do and what it actually does as models, prompts, tools and user behavior change. Their walkthrough of Microsoft Foundry frames observability as a loop of OpenTelemetry tracing, trace-linked evaluations, monitoring, optimization and red teaming. The central demonstration is an observe skill that can generate an evaluation dataset, run batch tests, optimize prompts, compare versions and roll back to the best-performing agent version from a sparse starting point.
Energy-Based Fine-Tuning Trains Language Models on Whole Responses
Microsoft Research’s presentation on energy-based fine-tuning argues that language-model post-training can be aimed at whole responses rather than next-token imitation. Carles Domingo-Enrich presents EBFT as a middle path between supervised fine-tuning and reinforcement learning: it samples model completions, compares them with ground-truth answers in a model-derived feature space, and turns that comparison into a policy-gradient update without a separate reward model or verifier. The reported results show gains over SFT on several coding and translation measures, with performance often comparable to RLVR while avoiding explicit correctness rewards.
Interwhen Verifies AI Agent Actions Before They Become Irreversible
Microsoft Research’s Amit Sharma presents Interwhen as a framework for moving AI agents from post-hoc checking to verified execution while they are still acting. The open-source library uses LLMs to turn natural-language instructions, policies, and partial responses into smaller verifiable properties, then applies symbolic or model-based verifiers to tool calls and intermediate behavior. Sharma argues that this lets agents continue normally when checks pass but interrupts them when a verifier detects a violation, addressing risks that final-output review may catch too late.
MagenticLite Brings Full Agent Workflows to Small Language Models
Microsoft Research is presenting MagenticLite as a full-stack agentic system designed to make small language models usable for multi-step work across a browser and local files. Weili Shi, Harkirat Behl and Hussein Mozannar argue that the capability comes from specializing the stack rather than relying on frontier-scale models: MagenticBrain handles planning, coding and delegation, while Fara 1.5 controls the browser. The release also emphasizes user oversight, with the agent pausing for credentials, approvals or other points where the user needs to take control.
GPT-Realtime-2 Turns Voice Agents Into Tool-Using Reasoning Systems
OpenAI’s Build Hour on GPT-Realtime-2 presented the new realtime voice release as a shift from conversational voice interfaces toward tool-using, stateful agents. Teri Yu and Erika Kettleson argued that GPT-realtime-2’s larger context window, stronger instruction following, parallel tool calling and controllable speech behavior let developers build voice systems that can operate apps, reason across workflows and know when not to speak. Sierra’s Ken Murphy and Soham Ray added that production voice agents still depend on the surrounding system: guardrails, tuned turn-taking, tracing, redaction, evaluations and customer-specific workflows.