Dan Fu
Assistant Professor of Computer Science and Engineering at UC San Diego and VP of Kernels at Together AI, focused on efficient machine learning systems, GPU kernels, hardware-aware algorithms, and scalable ML architectures.
Together AI Targets 100-Millisecond Responses With Full-Stack NVIDIA Inference
Together AI’s Dan Fu argues that low-latency inference is a stack-wide engineering problem, not a single model optimization. In NVIDIA’s account of the company’s work, Fu says Together AI uses NVIDIA GPUs and software including CUDA, CUTLASS, TensorRT-LLM and Dynamo to support projects such as a megakernel for returning the first 64 words of a voice-agent response within 100 milliseconds and ATLAS, a system for adapting speculative decoding as traffic changes.
Inference Constraints Are Reshaping Language Model Architecture
In a Stanford CS336 guest lecture, Dan Fu argued that language-model inference is no longer downstream plumbing but a central research and design constraint. Fu described serving as the machinery that turns a trained model into a usable system, where schedulers, KV caches, GPU kernels, routing policies and hardware choices determine which architectures are practical, economical and reliable at scale.