Inference and Deployment
Serving AI systems in production, including latency, scaling, observability, model routing, caching, edge deployment, and operational reliability.
A 91% Cache-Hit Rate Cuts One Agent Task From Over $200 to $21
George Cameron, co-founder of Artificial Analysis, argues that an AI agent’s cost depends on its full sequence of model calls, not just the model’s advertised token prices. In an AA-Briefcase benchmark example, a task averages $21 with a 91% cache-hit rate but costs more than $200 without caching. Cameron says teams should weigh cache hits, input volume, turns and reasoning effort, then benchmark models against the intelligence their own tasks require.
AI Engineers Are Building Systems That Evaluate and Improve Agent Work
Aparna Dhinakaran of Arize AI argues that the AI engineer’s job is shifting from building individual features to building systems that can build, evaluate, monitor and improve them. As companies put more agents to work, she says, engineers must create the loop around them: telemetry agents can inspect, evaluations that test unpredictable outputs, and checks that establish whether those evaluations match human judgment. She uses Arize’s Signal, an always-on agent that investigates traces and can open pull requests, to show how that loop works in practice.
Europe’s AI Strategy Depends on Building Local Compute and AI Factories
In his GTC Berlin keynote, NVIDIA CEO Jensen Huang argued that Europe’s AI economy will depend on producing intelligence, not just adopting AI services. He called for investment across the stack—from energy and computing to AI factories, models and applications—and presented NVIDIA’s hardware, software and partner ecosystem as infrastructure for that production. His case extended from industrial agents and robotics to secure runtimes and local workstations: AI, he said, requires systems built to run models, connect them to knowledge and tools, and put them to work.
GPU Contracts Assume Steady Demand; AI Workloads Do Not
Modal CEO Erik Bernhardsson argues that AI workloads need compute on demand, while much of the GPU market sells capacity in fixed, multiyear blocks that may begin months later. Modal pools supply across customers, regions and providers to match that capacity to demand that rises and falls. Bernhardsson says the model can make flexible access work, but leaves Modal exposed to the financial risk of long-term commitments if GPU prices or demand fall.
AI Demand Fluctuates, but GPU Capacity Still Sells in Three-Year Blocks
Modal CEO Erik Bernhardsson argues that AI workloads fluctuate too much for the GPU market’s prevailing model of capacity sold in fixed, multi-year blocks. In his account, Modal pools supply across customers and scales it with demand, but that flexibility requires the company to make increasingly large commitments to cloud providers—leaving it exposed if GPU prices fall or utilization weakens. The tension is central to how Bernhardsson weighs growth against the financial risks of securing capacity.
NVIDIA’s OpenShell Enforces Agent Access Outside the Model
NVIDIA’s OpenShell reference architecture argues that model alignment and instructions are not enough to secure autonomous agents: the systems around them must enforce limits on access. OpenShell applies operator-defined policies at the runtime boundary, mediating agents’ system, file and network access; an optional layer called Sentry monitors activity outside the host and can cut an agent’s connection to its model. NVIDIA presents the two controls as complementary safeguards for agents that use tools and data.
Pi Durable Resumes Interrupted Agent Work From Persistent Checkpoints
Alejandro Ao describes Pi Durable as a harness for agent applications that need to recover task state after crashes or interruptions, not as a coding-agent CLI. It stores conversations and checkpoints so work can resume when the host process starts again, while leaving tool replay safety and process restarts to the surrounding application. In a tutorial, Ao walks through the harness setup and demonstrates recovery with a local model and SQLite.
RTX 5090 Outruns M5 Ultra on Models That Fit in VRAM
A comparison of Apple’s M5 Ultra Mac Studio and NVIDIA’s RTX 5090 finds the graphics card faster on a Qwen3.8-27B test, leading in prompt processing and token generation. Its advantage depends on the model fitting within 32 GB of VRAM; the Mac’s 256 GB of unified memory can hold models that exceed that limit, making capacity—not benchmark speed—the deciding factor for larger local AI workloads.
Modal Has Run More Than 3 Billion Sandboxes for AI Agents
Modal CEO Erik Bernhardsson argues that the new capabilities of AI models require new infrastructure, including inference, training and sandboxes that let agents execute code in isolation. He describes those tools as building blocks for AI applications; Modal says it has run more than 3 billion customer sandboxes. The company has raised $750 million at a $15.75 billion post-money valuation to grow faster, launch products and add capacity.
Odysseus Separates the Agent Harness From the Model It Runs
PewDiePie’s Odysseus separates the agent application from the model it uses: the workspace coordinates tools and information, while Ajax, a fine-tuned Qwen 3.5 9B model, is intended to make tool-use decisions. In his breakdown, Alejandro AO argues that building a useful local agent depends not just on running software on your own machine, but on where the model runs and whether it can reliably use the harness’s tools. He explains how the agent loop works and why training on traces from the target setup may help a smaller model perform better.
Microsoft Plans Multi-Gigawatt Maia Deployment for Internal AI Workloads
Microsoft is moving its Maia chips into data centers as it seeks to run more of its AI workloads on homegrown hardware. Bloomberg reporter Matt Day said the company plans a multi-gigawatt deployment, with its third-generation Maia 300 beginning rollout in 2027, and expects internal services such as Copilot and GitHub to use the capacity. Microsoft’s case for the chips is that they can run some workloads more efficiently and reduce the cost of its data-center operations.
Private Document Processing Requires Controlling the Agent’s File Access
Filip Makraduli of Superlinked argues that sensitive document processing should happen before a coding agent sees the file: open models running on infrastructure the user controls can handle OCR, extraction and redaction, then return a smaller artifact for the agent to use. His Obsidian and MCP workflow is designed to reduce exposure to external model APIs and avoid spending context on raw documents, while letting teams choose or adapt models for specific tasks. But a private processing path does not alone guarantee privacy if the agent can read the original file first.
AI Gateways Let Teams Change Models Without Rebuilding Their Stack
As AI budgets face closer scrutiny, Remy Guercio of Tailscale argues that standardizing on one vendor is not a lasting answer to rising costs or AI lock-in. In his account, teams need room to change models, connectors, interfaces and execution environments without replacing the whole stack. An AI gateway, he says, can preserve that choice while giving organizations a central way to manage access, set model-specific budgets and inspect what agents do.
AI Agent Policies Need Runtime Enforcement at the API Gateway
Sam, CTO at Gravitee, argues that prompts and policy files cannot govern what an AI agent does once it can reach an API: enforcement has to sit in the infrastructure between the agent and the services it can call. In a live hotel-booking demonstration, he showed a gateway denying destructive actions and bulk access to guest data, while also limiting repeated requests, caching similar questions and rejecting oversized prompts before they reached a model.
LLM Design Trades Model Capacity Against Compute Per Token
Large language model design is a question of how to spend computation, not simply how many parameters a model has. In Stanford’s CME295 lecture, adjunct professor Afshine Amidi uses training budgets, sparse expert routing and attention mechanisms to show how designers allocate work across a model and its tokens; decoding choices, in turn, shape how the model generates answers. The distinction is between a model’s total capacity and the computation it uses for each token.
Codex Cloud Lets Developers Delegate Larger Tasks Without Managing More Machines
At DevDay 2026, Dominik Kundel argued that as developers delegate longer-running, more complex work to Codex, they should not have to keep a laptop open or provision additional machines to run it. He presented Codex Cloud, alongside the CLI, app and web interfaces, as a way to move tasks between environments while keeping their context and access to tools. The demonstrations also showed practical limits: cloud machines still constrain what can be built, and private services require explicit network configuration.
A Small Set of Choices Makes Model Decisions Nearly 10 Times Faster
OpenAI’s Decisions API is designed for applications that need a quick choice from a limited set of actions, rather than an open-ended response. In the launch demonstration, Romain Huet and Charlie Guo show how it turns text or image inputs into decisions an app can act on, such as routing sales leads or choosing a lane to steer a car. Huet says narrowing the model’s task to a few choices makes the API nearly 10 times faster.
Agent Improvement Depends on Linking Production Traces to Offline Evals
Marc Klingen, Langfuse’s co-founder, argues that improving agents is becoming less a matter of rewriting prompts and more a matter of building a feedback loop around them. AI can increasingly inspect production traces, propose fixes, and test them against datasets and evaluations, reducing work that teams once did manually. But Klingen says people still need to set the goals and judge what counts as a meaningful improvement; in a Langfuse changelog-writer demo, an agent identified internal jargon and tested a revision before it was submitted for review.
Goodput, Not FLOPS, Measures What Frontier AI Systems Actually Deliver
Amin Vahdat, Google’s chief technologist for AI infrastructure, argues that theoretical chip speed is a poor measure of AI capacity: at large scale, failures and recovery determine how much useful work a system delivers. He calls that measure goodput, and says it should guide decisions across the infrastructure stack, from chip design and networking to power and storage. As AI workloads shift toward long-running agents, Vahdat argues, the systems supporting them must evolve beyond accelerators alone.
The Scheduler Can Bottleneck Inference Without Doing the Heavy Compute
Charles Frye, who works on inference at Modal, argues that an inference engine’s central challenge is not generating tokens but serving them efficiently for the workload at hand. The scheduler can limit performance even though it does far less computation than the GPU, because it decides which requests reach the accelerator and how they are batched. Frye traces how workload demands, cache management and host-side overhead shape latency and throughput, and why operators need evidence from the deployed system to diagnose problems.
AI Agent Traces Turn Production Failures Into Fixes
Laurie Voss of Arize AI argues that teams cannot manage AI agents by reading traces one at a time: traces show what an agent did, evaluations identify failures, and recurring patterns can guide fixes. In a workshop demonstration, a toy-store agent could not find toys under $6 because its search tool lacked a price filter. Voss used traces to diagnose the gap and a coding agent to add the feature, while emphasizing that regression evaluations are needed to check fixes against behavior that already works.
Gemma 4 Brings Open Models to Browser and Mobile Devices
Paige Bailey of Google DeepMind argues that open models are becoming practical to run on ordinary hardware, including in a browser or on a phone, without sending data to a remote API. Her example is Gemma 4, a family ranging from 2 billion to 31 billion parameters that organizations can download, adapt and fine-tune; which version is useful depends on the device and its constraints.
Enterprises Are Building AI Strategies Around Model Independence
OpenRouter’s Alex Atallah and Replit CEO Amjad Masad argue that enterprise AI is likely to depend less on one all-purpose model and more on a mix of specialized systems that companies can choose, combine and control. Atallah makes the case for keeping model options open; Masad says businesses also need independence from model, cloud and data providers. Both see narrower models as potentially cheaper and easier to govern, though the speakers leave unresolved how agents should coordinate safely and when specialization beats a general model.
Pi Workspaces Turn Telegram Into a Gateway for Always-On AI Agents
Alejandro’s tutorial shows how to run specialized Pi assistants on an always-on machine and reach them through Telegram using his open-source `pi-gateway`. Each assistant runs from its own workspace, with its own instructions, tools and model; the gateway uses a local SQLite database to route messages to persistent Pi sessions. The demonstration covers a personal assistant and a research agent, while scheduled automations and support for other messaging platforms remain proposed extensions.