Inference and Deployment
Serving AI systems in production, including latency, scaling, observability, model routing, caching, edge deployment, and operational reliability.
Model Optionality Is the Defense Against Volatile Token Economics
Sarah Sachs, who leads AI engineering at Notion, argues that inference costs and rapid model changes can make an AI product economically untenable unless teams preserve the ability to switch suppliers. Because frontier labs are often both token vendors and direct product competitors, she says companies should route work by the cost, capability and latency a task requires—not by token price alone—and reserve frontier models for work that warrants them. Notion’s response is a multi-model architecture, with an auto model handling most traffic, alongside open-weight models, deterministic software and governance controls.
U.S. AI Restrictions Face an Enforcement Problem as Open Weights Spread
Cheap, capable Chinese AI models have turned model access into a conflict over national security, intellectual property and the economics of U.S. frontier labs, John Coogan argues. He and Jordi Hays say Washington may be able to target companies over alleged IP theft or infrastructure access, but open weights, fine-tunes and synthetic training data make it difficult to define—or enforce—a ban on a model’s Chinese origins. The result is a more immediate business question for users: which models are affordable, capable and permissible to deploy.
Nvidia Says Rubin Systems Are Shipping to Major AI Customers
Nvidia is responding to questions about its Vera Rubin rollout by arguing that the systems have moved beyond design and into deployment: major AI customers have received them, and at least one rack is running. Ian King reports that the company is also pitching robotic assembly and a claimed installation time of tens of minutes from a data-center loading dock to operation as evidence that its advantage lies in how quickly hardware becomes usable capacity.
GLM-5.2-Class Intelligence Could Reach a Single RTX 5090 by 2028
Ahmad Osman, founder of Osmantic, argues that rapid gains in model efficiency are bringing recently frontier-level capabilities onto hardware individuals and businesses can own. Citing the falling parameter requirements for comparable performance, he forecasts that GLM-5.2-class intelligence could run on a 32GB RTX 5090-class GPU by late 2027 or early 2028. For Osman, the consequence is strategic as well as technical: a GPU’s value rises as better open models become deployable locally, giving operators control over their models and access.
AI Security Scanning Shifts the Bottleneck to Verification and Patching
Anthropic’s Eugene Yan argues that frontier models have made vulnerability discovery abundant, shifting the practical security bottleneck to verifying, triaging, routing, and patching findings. Drawing on Mozilla’s jump in Firefox fixes and Anthropic’s scan of more than 1,000 open-source repositories, he proposes a six-step harness built around threat models, isolated sandboxes, independent exploit verification, and human-owned remediation. The aim is not to automate scanning alone, Yan says, but to turn model-generated candidates into reproducible, prioritized fixes.
AI Capability Depends on the Harness, Not Weights Alone
Daniel Han of Unsloth argues that AI capability is a property of the deployed system, not a model’s weights, context-window claim, or benchmark score alone. In his seminar, he says prompting, reasoning budgets, tool harnesses, serving configuration, numerical precision and verification rules can materially change results—and can create apparent gains when agents exploit the evaluator rather than solve the intended task. His practical conclusion is that teams should test the exact configuration they plan to deploy and treat benchmarks as attack surfaces as well as measurements.
Codex Expands From Code Generation to Software Work Coordination
OpenAI’s Romain Huet argues that Codex is evolving from a code-generation tool into a workspace for delegating and coordinating software work across planning, implementation, testing, deployment and pull-request review. The update centers on GPT-5.6 Sol and its higher-budget Ultra mode, which can split broad goals into parallel tasks, alongside visual browser and computer controls, inline edits, Sites deployment, and task management across desktop and mobile. Huet also notes that Ultra’s larger reasoning budget consumes token limits faster.
Long Context Cannot Replace Persistent Learned Memory
Engram co-founder and CEO Dan Biderman argues that larger context windows and retrieval systems cannot give AI the accumulated understanding needed to work reliably across an organization’s expanding knowledge base. Models that must reread documents, histories and decisions on every query are costly and prone to “context rot,” he says; Engram’s answer is a trainable memory layer that compresses recurring knowledge into learned representations. The longer-term challenge is deciding what a model should internalize, keep in explicit records and update over time without degrading performance.
Enterprises Are Shifting AI Spending From Token Volume to Outcomes
PwC TMT leader Dallas Dolen argues that enterprise AI spending should be judged by business outcomes, not by how many tokens employees consume or avoid. Speaking with Alex Kantrowitz at the Big Technology AI Summit, Dolen says companies need to benchmark AI costs, route work to the right models, and measure whether the output improves the product or service being delivered. His view is that falling model prices may give buyers more leverage, but they will not remove the need for discipline around cost, risk and employee trust.
DeepSeek Reports 60%–85% Faster Per-User Generation With DSpark
DeepSeek’s DSpark paper argues that large language models can be made faster at inference without making them more capable, by reducing waste in speculative decoding. Károly Zsolnai-Fehér presents the technique as a serving-side change: a smaller draft model proposes tokens, the larger model verifies them, and DSpark improves the process through short-range draft memory, early rejection of weak continuations, and confidence-based scheduling. The practical claim is a reported 60% to 85% per-user generation-speed gain over DeepSeek’s MTP-1 baseline, not the 661% throughput figure shown in exceptional cases.
Enterprise AI Buyers Are Turning Sovereignty Into a Vendor-Control Fight
The Palantir-Nvidia partnership is presented as evidence that enterprise AI safety is becoming a question of customer control rather than model access. David Sacks, Chamath Palihapitiya, David Friedberg and Jason Calacanis argue that companies and governments should not hand proprietary data, model weights, compute decisions and operating know-how to frontier labs that may later compete with them. The discussion extends from that AI sovereignty argument into a separate jobs dispute over whether current employment data can answer future displacement claims, and into fights over birthright citizenship and California’s budget as questions of institutional authority and fiscal accountability.
Meta’s Compute Sales Plan Exposes Its Missing AI Product Strategy
John Coogan and Jordi Hays argue on Diet TBPN that Meta’s reported plan to sell AI compute is less important as a potential cloud business than as a signal about its AI product strategy. Selling excess capacity could be a rational way to monetize a massive infrastructure buildout, they say, but it also raises a harder question: why Meta’s own apps are not yet producing enough obvious AI demand to use that compute internally. The debate turns on whether Meta Compute is a bridge to future products, a hedge, or evidence that the company has built ahead of its consumer AI strategy.
NVIDIA Recasts the Data Center as Infrastructure for Agentic AI
NVIDIA’s GTC Taipei recap argues that agentic AI will force a redesign of data center infrastructure around autonomous software loops rather than human-driven applications. The company frames Vera Rubin and the Vera CPU as systems built specifically for agent-scale workloads, while presenting DSX as a way for operators to extract more revenue from fixed power allocations. NVIDIA also casts Taiwan’s server manufacturing ecosystem, including Foxconn, Quanta, Wistron, ASUS, GIGABYTE, Pegatron and Wiwynn, as central to turning that architecture into deployable AI factories.
Etched Raises $800 Million to Challenge Nvidia in AI Inference
Etched chief executive Gavin Uberti told Bloomberg Technology the startup is positioning itself against Nvidia by treating AI inference as a rack-scale systems problem rather than a chip-only contest. The company, which has raised $800 million at a $5 billion post-money valuation from investors including Jane Street and TSMC-linked VentureTech Alliance, argues its low-voltage inference design and cluster-scale memory can lower cost per token as models get larger. Uberti said production scale is now the central test, though Etched did not disclose named customers, revenue, or comparative benchmark figures.
Together AI Targets 100-Millisecond Responses With Full-Stack NVIDIA Inference
Together AI’s Dan Fu argues that low-latency inference is a stack-wide engineering problem, not a single model optimization. In NVIDIA’s account of the company’s work, Fu says Together AI uses NVIDIA GPUs and software including CUDA, CUTLASS, TensorRT-LLM and Dynamo to support projects such as a megakernel for returning the first 64 words of a voice-agent response within 100 milliseconds and ATLAS, a system for adapting speculative decoding as traffic changes.
AI’s Scarce Inputs Are Rewriting the Open Versus Closed Model Debate
John Coogan, Jordi Hays and Tyler use Zhipu AI’s open-weight GLM-5.2 release to argue that the open-versus-closed AI fight is now about timing, not a settled winner. Closed labs may still lead at the frontier, but they say capable open models arriving close behind can weaken API-based controls, shorten monetization windows and complicate security planning. The discussion broadens that pressure into the AI supply chain, where scarce compute and memory capacity may be capturing profits before model providers can.
Cheaper Models and Restricted Access Are Weakening the Frontier AI IPO Story
Alex Kantrowitz and Ranjan Roy argue that frontier AI is entering a more constrained and less certain commercial phase, as Anthropic’s Mythos release and OpenAI’s limited GPT-5.6 preview make access to top models partly dependent on government-approved customer lists. Their discussion centers on the risk that gating, cheaper adequate models, routing tools, distillation concerns and billing scrutiny could weaken the premium-usage story behind OpenAI and Anthropic’s valuations. They also treat Apple’s broad price increases as less a clean pass-through of memory costs than an exercise of market power.
Small Local Models Can Replace Frontier Calls When Product Evals Prove Fit
RL Nabors of Arize argues that teams should stop treating frontier-model calls as the default for production AI features. Her case is to prototype with the strongest model when needed, then use golden datasets, capability evals, and trace-based measurements to work down to the smallest local or task-specific model that meets the product’s own bars for accuracy, latency, cost, privacy, and reliability. In her Mima summarization example, that process moved a Claude-backed feature to an on-device Llama 3.2 3B configuration, with evals becoming the guardrail for future changes.
Five Code-Level Fixes Cut Agent Token Costs Without Prompt Changes
Erik Hanchett, a developer advocate at AWS, argues that production agents can become expensive not because they fail, but because they repeatedly send too much to the model and use costly models for work that does not require them. His talk lays out five code-level controls for token spend: cache static prompts, route simpler tasks to cheaper models, keep large tool outputs out of repeated context, cap tool loops, and trim long conversation histories.
Cerebras Says Data-Center Capacity Is Its Main AI Growth Constraint
Cerebras chief executive Andrew Feldman told Bloomberg that investors are misreading the company’s post-earnings margin pressure, arguing it reflects a deliberate move to rent back sold equipment to meet demand rather than a weakening business. His broader case is that Cerebras is ahead of plan, avoids several AI chip supply bottlenecks through its architecture, and now faces a different constraint: the speed at which usable data-center capacity can be built.
The Cloud Is Being Rebuilt Around Agents, Tokens, and Sandboxes
Vercel chief executive Guillermo Rauch used a Stanford MS&E435 seminar to argue that coding agents are expanding the software market rather than merely making developers faster. In his view, AI is widening software creation from professional programmers to business users and autonomous agents, while shifting cloud demand from websites and applications toward deployed agents that need token routing, sandboxes, security, observability and long-running compute.
AI Engineering Is Moving From Model Benchmarks to Production Harnesses
Shawn “swyx” Wang argues that AI engineering is shifting from a race over raw model capability to the production systems that make models usable: evals, harnesses, memory, routing, infrastructure and auditability. Drawing on Cognition’s Frontier Code benchmark and his view of the AI Engineer agenda, Wang says the key software frontier is no longer whether agents can pass tests, but whether they can produce maintainable, mergeable code inside real organizations. His broader case is that unstable model access and enterprise constraints make the surrounding system, not the model alone, the durable product boundary.
Baseten Raises $1.5 Billion as Inference Demand Shifts Toward Open Source
Baseten’s $1.5 billion financing at a $13 billion valuation rests on a bet that AI inference is becoming a larger and more operationally demanding market as companies run more open-source and post-trained models. CEO Tuhin Srivastava says the capital will help Baseten secure diversified compute and build the infrastructure layer customers need, while Altimeter partner Apoorv Agrawal argues the shift is toward capability, control, and cost advantages rather than simple access to frontier models.
A $280 Single-Board Computer Can Replace a $28 Monthly Agent VPS
A video test of the Rubik Pi 3 argues that a small single-board computer can be enough to run an always-on AI agent server when the language model itself remains in the cloud. Comparing the Qualcomm Dragonwing-based board with a $28-a-month DigitalOcean VPS running OpenClaw, the presenter found that task times were often shaped less by hardware than by the agent framework’s planning, tool choices and execution path. His conclusion is that the Rubik Pi is not a replacement for fast interactive chatbots, but is viable for asynchronous background agents and could replace his VPS.