Vincent Koc

Vincent Koc is Lead AI Research Engineer and Developer Relations at Comet, where he works on AI/ML evaluation, LLM optimization, and developer education. He is a speaker and technologist with experience leading data and AI projects across sectors and presenting on generative AI, adaptive evaluation systems, and production-grade LLM workflows.

Fixed Evaluation Suites Go Stale as Agents Optimize Toward Intent

Vincent Koc of Comet ML argues that AI evaluation is being outpaced by the systems it is meant to measure. In a talk on adaptive evaluation for agents, Koc says static benchmarks and handcrafted test sets are poorly suited to applications that change with prompts, tools, production traces, user behavior and even their own harnesses. His proposed direction is to define the intended end state, use traces and telemetry to surface drift and edge cases, and treat evals as a continuously revised system rather than a one-time benchmark.

AI EngineerMay 12, 202611 min read