Inference and Deployment
Serving AI systems in production, including latency, scaling, observability, model routing, caching, edge deployment, and operational reliability.
Apple-Device AI Is Becoming Viable Without Cloud Inference
Prince Canuma presents MLX, Apple’s array framework for Apple Silicon, as a practical foundation for running AI agents locally rather than through cloud services. His case is rooted in accessibility and unreliable connectivity, but extends to product constraints for voice agents, robots and multimodal apps: vision, speech, video generation and long-context inference can increasingly run on Macs, iPhones and iPads without a network call. Canuma does not argue that local models replace every frontier cloud system, but that the boundary has moved far enough to make on-device AI a serious deployment option.
Durable Agents Need Context Logs and Execution Snapshots
Eric Allam of Trigger.dev argues that durable agents need more than the replay-based workflow model used for durable transactions. In his talk, he separates agent durability into two problems: the LLM context, which fits naturally as an append-only log, and the execution environment — files, memory, subprocesses and local state — which he says should be preserved through OS-level snapshot and restore. Allam uses Trigger.dev’s Firecracker work to make the case that long-running agents are becoming session-like workloads, not just replayable transactions.
Text-to-Speech Models Are Converging on LLM-Style Architectures
Samuel Humeau of Mistral argues that modern text-to-speech has converged on an architecture that resembles large language modeling: an autoregressive transformer generates compressed audio tokens frame by frame, rather than raw waveform samples. Using Mistral’s open-weight Voxtral TTS model as the example, he says neural audio codecs make that possible by reducing dense speech signals to token-like representations a transformer can handle. The remaining latency frontier, in his account, is not just streaming playable audio early, but letting TTS consume an LLM’s text stream as it is still being written.
Voice AI Still Confuses Natural Speech With Real Conversation
Neil Zeghidour, CEO of Gradium AI and one of the researchers behind the full-duplex voice model Moshi, argues that voice AI’s long-promised “Her” moment is still being confused with better synthetic speech. His case is that cascaded voice agents are useful but structurally too slow and lossy to feel conversational, while speech-to-speech models improve flow but remain limited unless they can listen and speak simultaneously, use tools reliably, understand paralinguistic cues, and run cheaply enough to scale.
Fresh Product Data Is the Constraint for LLM Commerce Discovery
Criteo executives Diarmuid Gill and Liva Ralaivola argue that modern ad tech is best understood as a millisecond-scale prediction system: anonymous commerce signals, learned embeddings and real-time auctions are used to decide whether to bid, what to show and how much an impression is worth. In a conversation with Nathan Labenz, they frame Criteo’s work with OpenAI and other generative tools as an extension of that problem, not a replacement for it: LLMs may change product discovery, but the system still depends on fresh retailer data, consent, latency discipline and human oversight.
Wayve Bets Licensed Onboard AI Can Scale Autonomous Driving
Wayve chief executive Alex Kendall tells Bloomberg that autonomous driving is shifting from hand-engineered, city-specific systems toward learned AI models that run onboard vehicles and improve from real-world driving data. His argument is also commercial: Wayve plans to license its autonomy platform to manufacturers and fleets rather than build cars or operate robotaxi networks, a model Kendall says can scale across more vehicles, sensor packages and driving environments.
Pretraining and Attention Infrastructure Made Vision Transformers Practical
Isaac Robinson of Roboflow argues that transformers overtook convolutional networks in vision not because images stopped needing visual structure, but because that structure moved from hand-built architecture into pretraining, scaling and tooling. In his account, ViT-style models first lacked the inductive biases and efficiency that made CNNs dominant, but self-supervised vision pretraining and attention infrastructure from the LLM world made the simpler architecture practical. Robinson frames the next problem as deployment: turning large foundation backbones into model families that can meet real latency, cost and hardware constraints.
BFL Is Moving FLUX From Image Generation Toward Physical AI
Stephen Batifol of Black Forest Labs argues that FLUX is no longer just an image-generation line but the start of a broader push toward visual intelligence: models that can generate, edit, understand, and eventually act across images, video, audio, and physical environments. In the talk, he presents FLUX.1, Kontext, FLUX.2, and FLUX.2 Klein as product steps toward that goal, while BFL’s Self-Flow research is framed as the mechanism for moving representation learning inside multimodal generative models rather than relying on external encoders.
Production Analytics Finds Agent Failures That Standard Evals Miss
Scott Clark, co-founder and chief executive of Distributional, argues that teams running LLM agents need to look beyond pre-production evals and dashboards of known metrics. His case is that the most consequential failures often emerge only in production, where agents interact with users, tools and changing models in ways teams did not know to test. Clark proposes an observability stack in which telemetry records what happened, monitoring tracks known signals, and analytics clusters trace behavior to surface unknown failure modes that can become new evals, guardrails, prompts or system fixes.
Production Agents Need Evals and Managed Variables After Deployment
Samuel Colvin of Pydantic argues that production agents need more than observability after deployment: they need evals, traces, and typed configuration that can change prompts, models, and other parameters without a redeploy. Using Pydantic AI, Logfire, managed variables, and GEPA, he shows a workflow for moving from manual prompt tuning toward continuous optimization. His case is practical rather than automatic: GEPA can improve a narrow benchmark, but only if the team has representative data, sound evaluation criteria, and a clear definition of what better means.
Production Agents Need Semantic Observability Beyond Offline Evals
Raindrop’s workshop argues that production agents need a different observability model from conventional software monitoring or offline evals. Zubin Kumar, Danny Gollapalli and Ben Hylak make the case that teams should track both explicit telemetry such as tool errors, latency and cost, and implicit signals such as user frustration, refusals, task failure, capability gaps and unusual workarounds. Their framework treats real production behavior as the primary surface for finding regressions, running experiments and catching failures that do not appear as clean exceptions.
Apple Explores Intel and Samsung for U.S. Chip Production
Mark Gurman said Apple has held early talks with Intel and Samsung about using new U.S. fabs to make future A-series and M-series processors, an exploratory move he framed as a supply-chain redundancy question rather than only a political one. Apple still relies heavily on TSMC, primarily in Taiwan, and Gurman described that geographic and supplier concentration as one of the company’s biggest risks. Across the rest of the broadcast, executives and analysts described a similar shift from exposure to execution: AI companies are giving Washington early model access for review, while enterprise adoption is being tested by security, deployment cost and proprietary data advantages.
Thoma Bravo Keeps AI Strategy Model Agnostic as Cyber Risks Accelerate
Thoma Bravo managing partner Seth Boro told Bloomberg’s Dani Burger that enterprise AI is creating parallel problems for companies: faster cyber threats and uncertain deployment economics. Boro said the firm is “model agnostic,” maintaining relationships with OpenAI, Anthropic and Google while using its cybersecurity portfolio to monitor emerging threats. He argued that enterprises will need layered defenses, tighter governance of AI agents and more specific, efficient models rather than assuming general-purpose systems fit every workflow.
DeepSeek V4 Claims Frontier-Adjacent Open Weights With One-Million-Token Context
Károly Zsolnai-Fehér of Two Minute Papers argues that DeepSeek V4 Preview is a consequential open-weight AI release because it pairs frontier-adjacent benchmark results with a reported one-million-token text context window and sharply lower long-context memory costs. His case rests less on outright benchmark dominance than on access economics: a freely self-hostable model appears close enough to recent closed frontier systems to change what developers can afford to use. He also stresses the limits: DeepSeek V4 is text-only, degrades near the edge of its context window, and still needs serious hardware at full scale.
Orbital Compute Becomes Cheaper If Launch Costs Fall Below $500/kg
Philip Johnston, Starcloud’s co-founder and chief executive, argues that AI data centers could become cheaper in orbit than on Earth if launch costs fall to about $500 per kilogram. His case rests on continuous solar power in a dawn-dusk orbit, avoiding land and battery costs, and using constellations of optically linked satellites for inference workloads. Starcloud’s plan, he said, starts with an orbital GPU proof point and points toward an 88,000-satellite network delivering roughly 20 gigawatts of compute capacity.
Small-Model Inference Needs Infrastructure Beyond Model Servers
Filip Makraduli of Superlinked argues that the hard part of small-model inference is no longer simply serving a model, but operating many embeddings, rerankers, extractors and multimodal models efficiently in production. In his account, conventional one-model-per-container deployments waste GPU capacity and leave teams to rebuild routing, autoscaling, monitoring, hot-swapping and eviction themselves. Superlinked’s SIE is presented as an open-source attempt to provide that missing infrastructure layer for AI search and document-processing workloads.
Enterprise AI Agents Need Harnesses, Traces, and Controlled Runtimes
LangChain co-founder and CEO Harrison Chase argues that enterprise AI agents are becoming an architectural problem rather than a question of adding autonomy wherever possible. In an NVIDIA AI Podcast interview, he says systems such as Claude Code, Manus and Deep Research share a common “deep agent” pattern: an LLM in a tool-calling loop, supported by a reusable harness, workspace, subagents and planning. For enterprises, Chase says trust depends on choosing the right level of autonomy and surrounding agents with observability, evaluation, secure runtimes and continued iteration.
Gemma 4 Moves On-Device AI From Chatbots to Local Agents
Chintan Parikh of Google DeepMind argues that on-device AI is moving from local chatbots toward local agents, as smaller Gemma 4 edge models become capable of tool calling, structured output and reasoning on phones, laptops and embedded hardware. With Weiyi Wang joining the Q&A, Parikh presents LiteRT as the deployment layer for that shift across Android, iOS, desktop, web and IoT. His case is pragmatic rather than absolute: edge inference can improve latency, privacy, offline use and cost, but teams still have to manage memory, quantization, accelerator support and when to call the cloud.