Charles Frye
Charles Frye is a Member of Technical Staff at Modal, where he writes and speaks about AI infrastructure, GPU workloads, model deployment, and production inference. He previously worked on Weights & Biases and holds a PhD from UC Berkeley.
The Scheduler Can Bottleneck Inference Without Doing the Heavy Compute
Charles Frye, who works on inference at Modal, argues that an inference engine’s central challenge is not generating tokens but serving them efficiently for the workload at hand. The scheduler can limit performance even though it does far less computation than the GPU, because it decides which requests reach the accelerator and how they are batched. Frye traces how workload demands, cache management and host-side overhead shape latency and throughput, and why operators need evidence from the deployed system to diagnose problems.
Production Inference Turns Transformer Models Into a Full-Stack Systems Problem
In a Stanford CS25 seminar, Modal’s Charles Frye argues that transformer inference has become the economic and operational center of AI systems: training produces weights, but serving turns them into usable, billable products. His account treats production inference as a full-stack problem, where application latency goals, workload shape, model choice, GPU memory limits, deployment failures, observability and cost controls all determine whether a system works. Frye’s main warning is that the largest serving gains come from matching the inference stack to the application, not from treating model hosting as a generic infrastructure task.