AI Engineer
Talks, workshops, events, and training for AI engineers.
Untied Ulysses Pushes Llama-3-8B Training to 5 Million Tokens
Together AI’s Max Ryabinin argues that training transformers at multi-million-token context lengths is chiefly a memory-scheduling problem, not a matter of applying a single long-context technique. Using a Llama 3-8B run on an 8xH100 node as the example, he shows how fully sharded data parallelism, DeepSpeed Ulysses, activation checkpointing, CPU offloading and chunked sequence training each remove one bottleneck and expose the next. His proposed addition, Untied Ulysses, chunks attention heads and reuses context-parallelism buffers, with the presented results claiming scaling to 5 million tokens with limited throughput loss.
Code Agents Need Context Engineering, Not Larger Prompts
Nupur Sharma of Qodo argues that larger context windows have not solved a core agent failure: models still tend to use the beginning and end of an input while losing important material in the middle. Her case is that agent quality depends less on giving a model more context than on engineering how context is retrieved, ranked, constrained and checked. She describes Qodo’s approach as a mix of iterative retrieval, specialist agents, judge nodes and bounded orchestration that reserves high-reasoning models for discovery while using stricter, lighter steps for validation.
Durable Objects and Dynamic Workers Reopen Eval for AI Agents
Cloudflare engineers Sunil Pai and Matt Carey argue that AI agents need compute primitives beyond stateless functions: Durable Objects for addressable, persistent coordination, and Dynamic Workers for safely running generated code. Pai frames Durable Objects as the execution unit behind Cloudflare’s Agents SDK, giving agents state, resumable streams, scheduling, and multi-client sync without pushing distributed-systems work onto developers. Carey and Pai present Dynamic Workers as the larger shift: a sandboxed “eval++” model where LLM- or user-generated code starts with no ambient authority and receives only explicitly granted capabilities.
Telemetry, Not Code, Audits Nondeterministic AI Agents
Dat Ngo of Arize argues that LLM observability has to account for failures in execution paths, not just broken components, because agents can call tools in different orders, branch, loop, and change behavior across runs. In his account, traces become the audit record for nondeterministic systems, while evaluation must combine model judges, human feedback, golden datasets, deterministic checks, and business metrics at the right scope. Arize’s stated direction is to connect observability, evals, experimentation, and improvement into an increasingly automated loop.
RunPod’s Serverless LLM Endpoint Trades Cold Starts for Lower Idle Cost
Audry Hsu presents RunPod as a cloud AI infrastructure company trying to move GPU provisioning and operations behind a deployable model endpoint. In the walkthrough, she shows a Qwen model deployed from RunPod’s Hub as an OpenAI-compatible vLLM serverless endpoint on H100s in under five minutes, with billing tied to workers while they handle requests. Her case is narrower than eliminating infrastructure tradeoffs: the first request waited 41.6 seconds on cold start, while subsequent execution took about 1.5 seconds, leaving teams to choose between lower idle cost and keeping workers warm for lower latency.
Agents Can Build and Repair Scrapers Instead of Parsing Every Page
Rafael Levi of Bright Data argues that the hard part of web data collection has moved from scraping a page to maintaining the pipeline after sites change. In his session, he presents Bright Data’s MCP, APIs and browser infrastructure as a way for agents to inspect public websites, generate reusable scrapers, run them at scale and repair them when selectors, pagination or access conditions break. The economic case is that LLMs should spend tokens learning site structure and writing code, not repeatedly parsing every page.
VS Code Can Render MCP Tool Results as Interactive Apps
GitHub’s Marlene Mhangami and Liam Hampton argue that MCP apps turn chat from a text response surface into a place where tool output can be operated directly. In their VS Code demo, an MCP server profiles a Go app, returns data plus a reference to a bundled HTML UI, and VS Code renders the result as a sandboxed interactive flame graph inside Copilot chat. Their case is that the useful boundary is precise: tools provide data, resources provide the interface, and the host contains the app while keeping the user in context.
Cline’s Terminal-Bench Gains Came From Harness Tuning, Not Model Switching
Ara Khan of Cline argues that AI evals are too noisy to treat as truth but too useful to replace with vibes. Using Cline’s Terminal-Bench work as the case study, he says the company’s jump from 43% to 57% came from harness changes — container CPU and memory, longer timeouts, and model-family-specific prompting — rather than a better model. His prescription is to run evals skeptically, inspect failed traces, allocate failures by cause, and improve only the levers that survive contact with product behavior.
Stripe Says Agent Payments Need Deterministic Controls, Not Browser Automation
Stripe’s Steve Kaliski argues that autonomous agents can use probabilistic reasoning to discover products, services and tools, but payments should move through deterministic infrastructure. In his talk, he presents Stripe’s approach to agent commerce: scoped payment credentials, HTTP-based paid tool calls and structured checkout APIs designed to prevent agents from paying the wrong merchant, buying the wrong item, authorizing the wrong amount or exposing the wrong credential.
OpenClaw’s 3,000-Commit Day Shows Code Review Becoming the Bottleneck
Vincent Koc uses OpenClaw’s high-velocity refactor to argue that agentic software development is becoming an industrial management problem, not a prompting trick. In his account, a project that briefly touched 82% of its core codebase and produced thousands of commits exposed a new bottleneck: the human ability to supervise parallel agents, trust the test harness, reject bloat, and stop sessions that have lost the plot.
Voice AI Benchmarks Understate Errors in Real Multi-Speaker Audio
Hervé Bredin of pyannoteAI argues that voice AI benchmarks often make speech-to-text look more solved than it is by evaluating cleaner, more single-speaker-like audio. In his talk, he shows Nvidia Parakeet scoring 11.4% word error rate on AMI meeting audio in the Open ASR Leaderboard but 26% in pyannoteAI’s run on the same dataset using the table microphone rather than headset audio. Bredin’s broader case is that conversational AI needs fine-grained speaker diarization and speaker-attributed transcription, because words alone do not capture who spoke, when they overlapped, or how real multi-speaker conversations are structured.
Text Diffusion Trades Batch Throughput for Faster, Revisable Generation
Google DeepMind’s Brendon Dillon argues that text diffusion changes language generation by refining blocks of tokens rather than committing to one token at a time. In his account, that gives diffusion models lower latency and the ability to revise earlier text after later reasoning emerges, but it also creates a serving problem: weaker throughput when many requests are batched at scale. Dillon frames the technology as most compelling today for on-device and interaction-heavy products, where fast, revisable generation matters more than large-batch economics.
AI Evaluation Is Falling Behind Agent Deployment in High-Stakes Domains
Vincent Chen of Snorkel AI argues that agent evaluation has not kept pace with the systems now being pushed toward real deployment. Drawing on more than 120 applications to Snorkel’s Open Benchmarks Grants, he lays out a framework for benchmarks that are rigorous enough to measure capability and opinionated enough to direct research. In Chen’s account, the next useful benchmarks will need validated tasks, intentional distributions, unsaturated headroom, and evaluation methods that capture realistic constraints, while also betting on richer environments, longer autonomy, and more complex outputs.
Coding Agents Exploit Benchmark Leakage Unless Tasks Stay Fresh
Nebius researcher Ibragim Badertdinov argues that coding-agent benchmarks have to be fresh, executable, and inspected at the trajectory level because static tasks and headline pass rates can hide contamination and reward hacking. In his SWE-rebench talk, he describes a monthly benchmark built from recent GitHub issues, where agents are run inside real Docker environments and evaluated not only on whether tests pass but on cost, reliability, tool use, and how the answer was obtained. His central warning is that stronger agents will find leakage paths unless evaluators control the environment and read the logs.
AI Engineering Must Preserve Craft as Work Shifts to Verification
At AI Engineer Melbourne, Jeremy Howard, Annie Vella and Mic Neale each argued against treating AI adoption as an automatic productivity upgrade. Howard warned that coding tools can simulate autonomy and flow while eroding mastery; Vella presented research showing engineers feel more productive even as parts of developer experience deteriorate; and Neale made the case for pooling idle edge devices as an alternative to defaulting all inference to centralized, metered infrastructure.
Declarative UI Is Emerging as the Practical Path for Agent Interfaces
Ruben Casas of Postman argues that agent interfaces have not caught up with the frontend code models can now generate. In his talk, he contrasts static component systems with declarative UI, where an LLM produces JSON or YAML for a renderer, and fully generative UI, where the model writes HTML, CSS and JavaScript directly. Casas says declarative UI is probably the right balance today, while MCP apps matter because their sandboxing offers a way to contain runtime-generated interfaces.
Semantic Search Cut Claude Code’s Wasted File Reads to One in Eight
Kuba Rogut of Turbopuffer benchmarked Claude Code on 50 ContextBench tasks to test whether it found the right code context, not whether it solved the tasks. He argues that adding semantic search to windowed grep made Claude Code’s file reads much more precise, cutting irrelevant reads from about one in three to one in eight, but did not make semantic retrieval a blanket replacement for grep. In Rogut’s results, semantic search helped when related code shared behavior rather than keywords, while grep remained stronger when the relevant term or import path was explicit.
BDD and ADRs Give AI Coding Agents Enforceable Project Memory
Michal Cichra of Safe Intelligence argues that AI-assisted development does not fail for lack of prompts so much as for lack of enforceable memory. In his talk, he makes the case for keeping ADRs, PRDs, BDD scenarios and design-system rules close to the code, so product intent and architectural decisions can be found by humans, retrieved by agents and enforced by Git hooks and CI. His most specific claim is that Cucumber-style executable specifications have become useful again because they connect human-readable product behavior to tests that prove the software still does what the spec says.
The Model Alone Is No Longer the AI Product
At AI Engineer Melbourne 2026’s Day 1 keynote program, speakers including Shawn Wang, George Cameron, Sarah Sachs, Igor Costa, Vamsi Ramakrishnan and Geoffrey Huntley argued that AI engineering has moved beyond picking the strongest model. Their shared case was that useful AI products now depend on the systems around models: harnesses, routing, evals, memory, state, latency budgets, deterministic tools and cost controls. The model still matters, but the keynote program framed product advantage as an architecture and economics problem, not a leaderboard problem.
AI Builders Are Urged to Architect the Future Through Early Adoption
At the Day 1 keynote livestream for AI Engineer Melbourne 2026, the opening speaker acknowledged the public debate over AI’s risks but argued that builders should not stop there. The speaker framed early adoption as a way to enter deeper conversations, form faster connections, and help “architect” the direction AI takes, with the conference itself presented as a participatory setting for that work.
Fine-Tuning Becomes the Next Step for Mature AI Products
Benjamin Cowen, a forward-deployed machine-learning engineer at Modal, argues that fine-tuning is becoming a normal stage in the maturation of AI products rather than a specialist research exercise. His case is that frontier APIs and product teams optimize for different goals: labs need broadly capable models, while companies need models that fit their own economics, latency constraints and business-specific quality metrics. Cowen says the decision point shows up when API costs overwhelm revenue, evals stop improving through prompting, or shared endpoints cannot meet throughput requirements.
High-Quality Agentic Tasks Drove 5x More Fine-Tuning Uplift
Snorkel’s Kobie Crawford argues that task quality, not just model size or compute, can determine whether agentic fine-tuning produces useful gains. In a Terminal-Bench-style experiment holding the base model, compute budget and task count constant, Snorkel reported that fine-tuning on rejected low-quality tasks improved Qwen3-8B by about one percentage point, while accepted high-quality tasks improved it by 6.2 points. Crawford’s case is that well-specified, reliable tasks create learnable failures, while ambiguous prompts, mismatched tests and broken environments mostly add noise.
Lovable Uses Agent Complaints to Find Bugs and Improve Projects
Benjamin Verbeek of Lovable argues that AI coding products can improve continuously by treating user failures and agent frustration as production signals. In a talk on Lovable’s internal systems, he describes two loops: one that turns sessions where nontechnical users get stuck and later recover into tested contextual guidance, and another that lets the agent complain directly when Lovable’s tools, documentation or platform behavior block its work. Verbeek says the approach has surfaced real bugs, reduced repeated “fix” intent messages and created an operational signal for incidents.
State-of-the-Art AI Models Are a Pareto Frontier, Not a Ranking
Bertrand Charpentier, cofounder and chief scientist at Pruna AI, argues that state-of-the-art image generation should not be defined by a single leaderboard rank. Using Design Arena-style evaluation as his example, he says a slow top model can require 20 days of compute, about $5,300 and 556 kWh to evaluate, while a fast compressed model can run the same test in 7 hours for $265. His broader case is that model selection should be based on a Pareto frontier of quality, latency, cost and energy, not a podium that treats efficiency as secondary.