Inference and Deployment
Serving AI systems in production, including latency, scaling, observability, model routing, caching, edge deployment, and operational reliability.
DeepSeek’s DualPath Raises GPU Utilization by Rerouting KV-Cache Traffic
Károly Zsolnai-Fehér presents DeepSeek’s DualPath paper as an infrastructure fix for a specific bottleneck in agentic LLM serving: GPUs can sit underused because KV-cache data cannot reach the model fast enough. The work, from researchers affiliated with Peking University, Tsinghua University and DeepSeek-AI, argues that routing memory traffic through underused decoding machines can relieve the congested prefill path while keeping computation traffic prioritized. In the demonstrated setting, Zsolnai-Fehér says GPU utilization rises from about 40% to about 80% without adding compute.
World’s Fair 2026 Recasts AI Engineering Around Deployment and Operations
Shawn Wang is pitching AI Engineer World’s Fair 2026 as more than a larger version of the group’s past conferences. In a video update, he argues that the San Francisco event is being redesigned around how AI engineering work is changing: a much larger expo floor, closer links between research and shipping teams, high-volume token operations, AI leadership practice, and vertical deployment in sectors such as finance and healthcare.
AI’s Next Bottleneck Is Compute Waste, Not GPU Scarcity
Anjney Midha, AMP’s founder and an investor in frontier AI companies including Anthropic and Mistral, argues that AI’s infrastructure bottleneck is as much waste and misalignment as GPU scarcity. In a conversation with swyx at Periodic Labs, he makes the case for AMP as a neutral compute grid that would pool supply and demand so FLOPs can move more like megawatts. Midha ties that infrastructure thesis to a broader discipline he calls “output maxing”: raising utilization, reducing organizational loss, earning community trust for data centers, and making frontier systems deliver more useful work from scarce resources.
Power and Heat Are the Hard Limits for Orbital AI Data Centers
Makenzie Lystrup, a principal consultant at Peridot Services and former director of NASA’s Goddard Space Flight Center, argues that orbital data centers should not be treated as one idea. In a Bloomberg Technology interview, she says near-term edge computing in orbit is plausible, while hyperscale AI infrastructure of the kind associated with SpaceX faces much harder constraints: power systems, heat rejection, radiation-tolerant hardware, networking, reliability and maintenance. Her central point is that the challenge is not merely launching servers into space, but operating them as space-qualified infrastructure.
Tokens Can Now Substitute for 100-Person Startup Engineering Teams
In a Stanford CS153 lecture, OpenAI chief executive Sam Altman argued that AI has already rewritten the startup playbook, allowing small teams to buy capabilities with tokens that once required large engineering organizations. He used OpenAI’s experience with ChatGPT, Codex and model scaling to make a broader case: scale keeps producing capabilities that experts underestimate, but the institutions around AI — from education and research pipelines to compute markets and governance — are not adapting as quickly. Altman said the central choice ahead is whether intelligence becomes a broadly available utility or remains concentrated in a few companies.
MiniCPM-V 2.6 Runs at 18 Tokens per Second on iPhone
OpenBMB used its Build Small hackathon session to argue that small models are valuable when they can be deployed where applications and data already live: on phones, laptops, mobile apps and edge devices. Its main example was MiniCPM-V 2.6, a vision-language model shown running on an iPhone 15 Pro at 18 tokens per second with llama.cpp and 4-bit quantization. The broader claim was that compact, open models paired with existing runtimes can expand access, reduce cloud dependence, and improve privacy and latency for local AI use cases.
Apple’s New Siri Tests Who Controls the Default AI Assistant
John Coogan and Jordi Hays read Apple’s WWDC as a test of whether the company can turn its long-delayed Siri promise into a defensible AI interface without giving up control of defaults, privacy, and the iPhone camera. The Diet TBPN segment argues that Apple’s AI story is less about a single keynote than about older bets now becoming technically possible, while Anthropic’s Claude Fable release and Meta’s data-center training push show the same shift toward long-running inference and physical AI infrastructure.
A Python Decorator Replaces the GPU Deployment Container Loop
RunPod’s Audrey Hsu argues that GPU inference development should not require a commit, container build, registry push and server provisioning cycle for every model change. In a demo of Flash, RunPod’s Python SDK, she shows how adding a `@flash.endpoint` decorator to an async function can package that function as a GPU-backed cloud endpoint while the rest of the application stays in the developer’s IDE. Her broader case is that teams should experiment on Pods or low worker counts, then move to Serverless when they need autoscaling inference across many GPU workers.
Apple’s AI Challenge Shifts From Invention to iPhone Integration
John Coogan used Diet TBPN’s WWDC discussion to argue that Apple’s AI challenge is now less about inventing a breakthrough than deciding how deeply Siri, iOS, third-party models and cloud inference can touch the iPhone without breaking Apple’s privacy and product-control instincts. The episode also framed strong US hiring as a problem for tech’s rate-cut hopes, and separated viral VC pitch-room complaints from the more serious risk of opaque financing structures that founders may misrepresent.
Tech’s Hard Problems Are Moving From Demos to Deployment
TBPN’s Jordi Hays and John Coogan use Apple’s WWDC, the jobs report, venture-capital disputes, and interviews with operators in satellites, biotech, fusion, robotics and nuclear power to frame a recurring divide between demonstration and deployment. Their argument is that AI features, reactors, robots, medicines and market stories are now being judged less by whether they can be shown than by whether they can be operated at scale, with infrastructure, regulation, capital and user trust doing much of the hard work.
Durable Objects and Dynamic Workers Reopen Eval for AI Agents
Cloudflare engineers Sunil Pai and Matt Carey argue that AI agents need compute primitives beyond stateless functions: Durable Objects for addressable, persistent coordination, and Dynamic Workers for safely running generated code. Pai frames Durable Objects as the execution unit behind Cloudflare’s Agents SDK, giving agents state, resumable streams, scheduling, and multi-client sync without pushing distributed-systems work onto developers. Carey and Pai present Dynamic Workers as the larger shift: a sandboxed “eval++” model where LLM- or user-generated code starts with no ambient authority and receives only explicitly granted capabilities.
OpenAI Pitches Frontier AI as Infrastructure for Financial Services
Katy Elkin, OpenAI’s go-to-market lead for financial services, argues that banks, insurers, asset managers and market-infrastructure firms should treat frontier AI as enterprise infrastructure rather than a set of isolated tools. Her case is that financial institutions can use OpenAI’s models to redesign workflows, increase employee output and build AI-native customer products, provided they also put in place the governance, security and residency controls needed to absorb rapid model improvements.
Telemetry, Not Code, Audits Nondeterministic AI Agents
Dat Ngo of Arize argues that LLM observability has to account for failures in execution paths, not just broken components, because agents can call tools in different orders, branch, loop, and change behavior across runs. In his account, traces become the audit record for nondeterministic systems, while evaluation must combine model judges, human feedback, golden datasets, deterministic checks, and business metrics at the right scope. Arize’s stated direction is to connect observability, evals, experimentation, and improvement into an increasingly automated loop.
Sanders’ 50% AI Stock Plan Turns Training Data Into a Political Fight
Jason Calacanis argued that Anthropic’s call for an AI slowdown and Bernie Sanders’ proposal for public ownership of major AI companies show AI politics moving toward jobs, ownership and redistribution. He dismissed Sanders’ 50% stock-tax plan as unworkable but said its premise could resonate with voters who believe AI companies built enormous value from public and creative inputs while threatening employment. Yoland Yan’s ComfyUI demo supplied the production-layer version of the same control question, presenting generative AI as a workflow where exposed parameters and reproducibility matter more than prompt-box convenience.
RunPod’s Serverless LLM Endpoint Trades Cold Starts for Lower Idle Cost
Audry Hsu presents RunPod as a cloud AI infrastructure company trying to move GPU provisioning and operations behind a deployable model endpoint. In the walkthrough, she shows a Qwen model deployed from RunPod’s Hub as an OpenAI-compatible vLLM serverless endpoint on H100s in under five minutes, with billing tied to workers while they handle requests. Her case is narrower than eliminating infrastructure tradeoffs: the first request waited 41.6 seconds on cold start, while subsequent execution took about 1.5 seconds, leaving teams to choose between lower idle cost and keeping workers warm for lower latency.
Tech Founders Argue IPOs Can Create More Upside After Listing
At an All-In Liquidity IPO panel, Altimeter’s Brad Gerstner, Cerebras chief executive Andrew Feldman and Planet Labs chief executive Will Marshall made the case that public markets are again becoming a place where venture-backed technology companies can compound, not merely exit. Gerstner argued that investors often give up large gains by forcing distributions after an IPO, while Feldman said more money is historically made after companies go public than before. Marshall and Feldman also described the IPO less as an operating transformation than as a change in capital, credibility and scrutiny, with execution still determining whether the listing creates lasting value.
AI Application Companies Are Moving Beyond Frontier APIs to Protect Margins
Baseten founder and chief executive Tuhin Srivastava used a Stanford MS&E435 seminar with instructor Apoorv Agrawal to argue that inference is becoming the cost of goods sold for AI applications. His case is that scaled AI companies will need to move beyond default frontier-model APIs toward custom or post-trained models, both to improve margins and to protect the workflows and user signals that make their products defensible. Baseten’s role, as Srivastava framed it, is to provide the production inference stack and compute access needed to run that custom intelligence at scale.
Inference Constraints Are Reshaping Language Model Architecture
In a Stanford CS336 guest lecture, Dan Fu argued that language-model inference is no longer downstream plumbing but a central research and design constraint. Fu described serving as the machinery that turns a trained model into a usable system, where schedulers, KV caches, GPU kernels, routing policies and hardware choices determine which architectures are practical, economical and reliable at scale.
AI Infrastructure Is Shifting From Accelerator Racks to Distributed Agent Systems
At Dell Technologies World, Nvidia chief Jensen Huang and Dell CEO Michael Dell argued that enterprise AI is moving from experimental promise to operational infrastructure, with agentic systems driving a sharp increase in compute demand. Huang said agents change the workload from single prompt-response transactions to long-running loops of reasoning, planning and tool use, while Dell framed the response as a pragmatic push toward distributed, “unmetered” intelligence across PCs, data centers and cloud-scale systems.
Starcloud Shifts Orbital AI Compute Plan Toward 88,000 Inference Satellites
Bloomberg’s Ed Ludlow and Starcloud chief executive Philip Johnston frame orbital data centers less as cloud facilities moved off Earth than as specialized spacecraft built around compute, power, communications, flight systems and heat rejection. Against SpaceX’s stated ambition to deploy 100 gigawatts of AI compute capacity in orbit, Johnston argues that the nearer-term architecture is likely to be distributed inference satellites, not giant training platforms, with Starcloud filing for an 88,000-node constellation while starting from a single satellite carrying five GPUs.
Hackathon Caps Models at 32B Parameters to Reward Tinkerable AI Apps
Build Small is a Hugging Face and Gradio hackathon organized around a hard constraint: every model used must be under 32 billion parameters. Yuvraj Sharma framed the rule as a way to move AI building away from dependence on giant hosted models and back toward systems that participants can inspect, fine-tune, run locally, and ship as working Gradio Spaces. Sponsor presentations from Black Forest Labs, OpenBMB, OpenAI, NVIDIA, Modal, JetBrains, and Cohere largely reinforced that premise, offering small models, credits, tools, and prize categories meant to turn the constraint into runnable projects rather than demos in name only.
Production Inference Turns Transformer Models Into a Full-Stack Systems Problem
In a Stanford CS25 seminar, Modal’s Charles Frye argues that transformer inference has become the economic and operational center of AI systems: training produces weights, but serving turns them into usable, billable products. His account treats production inference as a full-stack problem, where application latency goals, workload shape, model choice, GPU memory limits, deployment failures, observability and cost controls all determine whether a system works. Frye’s main warning is that the largest serving gains come from matching the inference stack to the application, not from treating model hosting as a generic infrastructure task.
Enterprise AI’s Constraint Is Judgment, Not Token Consumption
At TBPN’s AIPCon 10 broadcast, Palantir chief executive Alex Karp argued that enterprise AI’s central problem is no longer model capability but organizational judgment: companies are consuming tokens, dashboards and AI-generated artifacts without tying them to decisions that change operations. AIG’s Peter Zaffino, Palantir’s Chad Wahlquist and USDA’s Sam Berry extended the same case from insurance, deployment architecture and government data systems, describing AI as valuable only when embedded in workflows, data structures and feedback loops that reflect how institutions actually work.
Text Diffusion Trades Batch Throughput for Faster, Revisable Generation
Google DeepMind’s Brendon Dillon argues that text diffusion changes language generation by refining blocks of tokens rather than committing to one token at a time. In his account, that gives diffusion models lower latency and the ability to revise earlier text after later reasoning emerges, but it also creates a serving problem: weaker throughput when many requests are batched at scale. Dillon frames the technology as most compelling today for on-device and interaction-heavy products, where fast, revisable generation matters more than large-batch economics.