Orply.

Local AI Performance Depends on Matching Each Workload to Its Stack

AI EngineerSaturday, November 21, 202616 min read

At CODE Summit 2026, Osmantic founder Ahmad Osman argued that local AI performance depends on matching each stage of a workload to the right model, hardware and inference engine—not choosing one stack for an entire application. He said teams should test the complete configuration against real tasks and operating constraints, including latency, concurrency, privacy and cost, and give each stage a tool, a test and a fallback. Osman also made the case that machines under users’ control could eventually support continual learning from their own outcomes.

Place the stage, not the application

? ahmad-osman’s central design rule is to decide where each stage of a workload should run, rather than choosing one model or machine for an entire application. A support email, for example, might be classified by a small model, checked against billing records by code, drafted from those records by a mid-sized model, and sent to a stronger model only if a bug needs investigation. A scoped tool and a person handle the refund itself. Each handoff can have a test; when a test fails, the system stops, asks a person, or escalates.

That separation matters because the stages have different requirements. Classification needs to return a label reliably. Retrieval needs to find the right account and policy. A draft must get its numbers from the records. A refund is an action, not a prediction: code should limit what can be done, and a person should approve it. Osman’s example is illustrative, but it captures his broader point: frontier intelligence is useful where a step is hard, not necessarily at every step.

A local system, in this view, is not synonymous with a model running on the user’s laptop. It could mean a device that works offline, a box the organization runs, weights on rented hardware, or a provider’s API. The practical questions are who runs the model, where it runs, and what data crosses the connection. “Local” describes a relationship to a machine or operator, not a guarantee that data stays on a particular device.

Osman connects that placement decision to the workload’s actual constraints: task quality, response time, context length, number of users, privacy rules, and the cost of running the system. One demonstration compared a local Qwen 3.6 27B model on an RTX PRO 6000 with GPT-5.5 in the cloud. Asked how a transformer generates a token, the local setup displayed a decode rate of 51 tokens per second and a total time of five seconds. The displayed GPT-5.5 figures were 34 tokens per second and 12.2 seconds overall. Osman used this single request and configuration to show that a local machine can win on a particular task; the comparison does not establish a general ranking of local and hosted models.

That is not a general verdict on model quality. Osman cited a substantial gap between the leading closed- and open-weight models on Artificial Analysis’s index. On other tests, however, the gap ranged from 43 points in favour of the best closed model on HLE Diamond to a two-point open-model lead on Terminal-Bench. He said he did not know how the evaluation problem would be solved. Which benchmark you trust can change the apparent distance between open and closed models; a broad score may also have little resemblance to the job you need done.

He pointed to a difference between traffic and spending. In OpenRouter data for the week of October 19, 2026, open-weight models accounted for 70.3% of paid tokens but 27.5% of paid spend. Across several task categories, the most-used model by tokens differed from the one receiving the most spend. Osman’s interpretation is a two-tier pattern: cheaper models do much of the volume, while users pay more for selected hard cases.

70.3%
of OpenRouter paid tokens were on open-weight models in the week of October 19, 2026
OpenRouter paid trafficClosed weightsOpen weights
Share of paid tokens29.7%70.3%
Share of paid spend72.5%27.5%
OpenRouter paid traffic for the week of October 19, 2026; free variants excluded.

This is why his advice is to measure the task, not to crown a model. For a given workflow, that means checking task accuracy at the exact model, quantization, runtime and hardware configuration you intend to deploy. It also means deciding what happens when the model misses: low confidence can go to a person; missing evidence can trigger a question instead of a draft; an unsafe action can require approval.

A capable model still needs its tools and harness

A failed task does not necessarily mean the model is too small. Osman described a friend’s RTX 5090 running Qwen 3.5 27B through Claude Code. The request was to change the GPU’s RGB colors. The model failed. Osman asked whether it had a search engine. It did not. When they repeated the request with a search endpoint configured through an environment variable, the same model and harness succeeded.

The point was not that search solves every local-agent failure. Osman described one request on one setup. The model had been asked to use a tool whose relevant information might not have been in its training data. Claude Code could search for information about the GPU and the steps needed to control it; the initial setup could not. Before buying a larger GPU or model, Osman argued, check whether the system has the tool the task requires.

That system includes more than weights. Osman described four parts of an inference engine: a traffic cop that decides which request runs when; a memory manager for weights and KV cache; a kernel dispatcher that selects the code performing the math; and a scheduler and API for batching and serving. The same model on the same hardware can behave like a different machine under another engine. In a coding-benchmark pilot shown in the talk, a standard agent scored 50% with Claude Opus 4.7, 30% with GPT-5.5 and 40% with Gemini 3.1 Pro. Their makers’ agents scored 40% with Claude Code, 40% with Codex CLI and 20% with Gemini CLI, respectively. The accompanying token counts also differed. These results show why comparisons need a fixed harness: otherwise the benchmark may be measuring the surrounding system as much as the model.

The video-editing trailer Osman showed made that distinction concrete. He said Claude Opus 5.5 orchestrated the work: it found clips and music, proposed overlays and assembled the edit. But the job did not run on one machine. Two RTX PRO 6000 GPUs handled image tagging, video upscaling and audio cleanup; a MacBook ran FFmpeg, rendered overlays in headless Chrome and performed the final encode; Osman made A/B selections and checked the result. A later 17-minute render took about nine minutes on the GPUs and eight on the MacBook. Moving the Mac’s work was estimated to save two to four minutes, which Osman judged not worth the added setup.

The models changed with the defect, too. In the A/B tests he described, SeedVR2-7B won all five video-upscaling comparisons, though it was roughly six times slower per clip than Real-ESRGAN. DeepFilterNet3 won four of five audio cases; MossFormer2 won one noisy clip. Osman cut Resemble Enhance because it could change the words. His rule: pick a model for the specific problem, and accept slower processing when nobody is waiting.

This kind of composition still needs operational discipline. A model’s score of 0.99 is not permission to issue a refund, prove a duplicate charge, or send customer data to a hosted service. Osman’s example treated the score as a possible routing signal, provided its threshold had been tested on the organization’s own labelled cases. The ledger establishes whether a charge was duplicated; scoped code limits the action; policy governs whether data can leave. A local model also does not guarantee a private workflow: a hosted drafting stage may receive the email and account record even if classification and retrieval stay inside the organization. Osman’s proposed mitigation was to have a local model replace names and IDs with placeholders before sending a draft request to a hosted model, then restore them afterward. He cautioned that masking names does not remove patterns in the data, so teams need to inspect the payload that actually leaves.

A stage is not ready for placement just because a model appears to work in a demo. Osman’s example acceptance tests ask whether held-out classification accuracy and p99 latency clear the team’s bar; whether retrieval returns the right account and incident; whether every number in a draft matches the database; whether a fix passes a test that reproduces the bug; and whether the account shows exactly one refund. Failures need defined fallbacks: low confidence goes to a person, missing evidence prompts a question, a weak draft escalates, and a human approves every refund. As he put it:

A stage isn't placed until it has a test and a fallback.
? ahmad-osman · Source

Owning a machine does not make it reliable by itself. Osman described a render pipeline that failed because of a driver mismatch, a full disk, an incorrect CUDA dependency and a script that printed “DONE” after FFmpeg had died. In the last case, a pipe hid the failed exit code. The remedy is to check the real exit status, expected output streams and duration, and the rendered artifact itself. His broader checklist was to pin the model and file revision, tokenizer and chat template, tool schema and parser, runtime and kernels, GPU, context, concurrency, precision and sampling settings. Acceptance should include task cases and latency under load, plus a fallback and rollback plan. When one part changes, rerun the checks that depend on it.

The workload’s bottleneck determines performance

Choosing hardware begins with a distinction Osman said buyers often overlook: prefill and decode have different bottlenecks. Prefill reads the prompt and builds the KV cache before the first token appears. Decode then generates the answer one token at a time. A long prompt can increase the wait for the first token; the speed of generating the answer is a separate measure.

For a dense model serving one request at a time, decode is often limited by memory bandwidth. The system has to read the weights for each generated token. A machine may have enough memory to hold a large model but still read it slowly. Osman’s comparison used the same 13.5 GB model size: a DGX Spark with 128 GB of memory and 273 GB/s of bandwidth had an idealized ceiling of about 20 tokens per second, against roughly 133 for an RTX 5090 with 32 GB and 1,792 GB/s. Capacity determines whether the model fits; bandwidth helps determine how quickly it talks.

Long context and multiple simultaneous users add another bill: KV cache. In one worked example, a model with 32 layers, eight KV heads and a 128-wide head dimension used 1 GiB of FP16 cache per user at 8K context and 4 GiB at 32K. Four users at 32K would therefore use 16 GiB for cache alone. A machine that fits the weights for one request may not fit the weights, runtime and cache for several long-running requests. Osman’s warning was to size for users multiplied by context, not parameter count alone.

Even an advertised context window does not tell you how much context is practically useful. Retrieval results shown in the talk fell as context length grew across several models. For a document collection, Osman also compared full-context prompting, retrieval-augmented generation and a memory-module approach. In a reported test on 512 documents, the module used 16 memory tokens and took 0.74 seconds, compared with 67,385 tokens and 19.9 seconds for full context. He noted that it could lose exact names; exact facts still belonged in the database.

Quantization can reduce the memory needed for weights, but “4-bit” is not a complete specification. A quantized file may use a different recipe for different layers, and weights, activations and KV cache can each be represented differently. Osman emphasized that the rounding error is not equally costly everywhere. Errors in some tensors can carry through later computation; exact strings, tool arguments and long reasoning may suffer before other capabilities. A uniformly low-bit file can damage a model even when its size looks attractive.

He distinguished quantization-aware training from post-training quantization. In quantization-aware training, training simulates rounding so weights can adapt to the precision they will eventually use. The resulting weights can then be converted to a real low-bit file and served with the same sort of kernels as a post-training-quantized file. Another approach, calibration-based post-training quantization, measures activation patterns and assigns precision tensor by tensor. Osman warned against reading the word “dynamic” as a single method: it can mean activation scales calculated at runtime, or a file recipe that assigns fixed precision to different tensors before serving.

The practical test is not whether a file is labelled 2-bit or 4-bit, but whether it passes the intended job. Osman distinguished top-token agreement with a full-precision model from task success: agreement can help rank quantized files, but only an evaluation of the task can establish whether a file works. The model, file, runtime, GPU, prompt format and tool schema belong to one tested configuration.

A live hardware comparison illustrated why both software and workload matter. On one request, the same Qwen 3.6 27B model ran at 51 tokens per second on an RTX PRO 6000, 23 on an M5 MacBook Pro, 13 on a Strix Halo and 11 on a DGX Spark. The stacks differed: the tower used vLLM and FP8, while the other machines used llama.cpp and Q4_K_M. The numbers describe complete configurations, not silicon in isolation.

The 50-request burst made the difference between those configurations more pronounced. The tower processed the set in 4.1 seconds, at 1,291 tokens per second overall. The MacBook took 158.3 seconds, the Strix Halo 154.0 seconds and the DGX Spark 158.3 seconds; each delivered about 32 tokens per second in total. The runs produced roughly 5,000 tokens per machine. This compares whole stacks: the tower differed in engine, precision, kernels and GPU. The test shows how those configurations served the requests, not which single component caused the gap. Osman attributed the weaker Mac result to a build that did not use its matrix-multiplication hardware for batching.

Machine and stackOne user (tokens/s)Fifty users (tokens/s total)Time for 50 answers
RTX PRO 6000 tower, vLLM / FP8511,2914.1 s
M5 MacBook Pro, llama.cpp / Q4_K_M2332158.3 s
Strix Halo, llama.cpp / Q4_K_M1332154.0 s
DGX Spark, llama.cpp / Q4_K_M1132158.3 s
Qwen 3.6 27B runs in a one-user comparison and a 50-request burst. These are whole-stack results, not an isolated comparison of hardware.

Batching changes the economics of a GPU because one read of the weights can serve several users’ tokens, allowing more of the compute to be used. That matters for agents, which often wait on shell commands and tool results rather than spending all their time generating tokens. But batching introduces scheduling choices: a long prompt can delay other users’ decode, and an agent team creates multiple concurrent requests plus extra prefill from messages between agents.

Osman’s view is that no single inference engine should be expected to optimize every model and chip. He listed portable llama.cpp, Mac-oriented MLX, CUDA-focused ExLlamaV3 and production-serving systems such as vLLM, SGLang and TensorRT-LLM, each intended for different situations. A general engine can support many devices, but platform-specific kernels and scheduling can matter. Osman also noted that the OpenRouter results he showed for GLM-5.3 varied by host: the best and worst listed GPQA Diamond scores were separated by six points. The result, he argued, is another reason to measure the endpoint and configuration rather than rely on a model name.

He urged engineers to experiment with small kernels and contribute optimizations for the hardware and models they actually use. For buyers, his proposed measurements were concrete: time to first token at the real prompt length; tokens per second per user at peak concurrency; requests per second that meet the p99 latency target; and task accuracy at the exact quantization and runtime. A specification sheet cannot answer those questions. Replay a representative day of requests instead. He also cautioned that a faster component may barely change the end-to-end request: in one illustrative calculation, making a tokenizer 100 times faster produced an 11% overall speedup when the tokenizer accounted for 10% of the request time.

Own steady work; rent bursts when the numbers favour it

Owning a GPU is not automatically cheaper than using an API. Osman’s cost model makes utilization central: an API charges per token, while a GPU costs money by the hour whether busy or idle. In one illustrative calculation using a $2-per-hour GPU and a measured burst rate, the estimated GPU cost varied by 100 times between 1% and full utilization. The local and hosted models in the comparison differed in quality and capability, so the figure was a GPU cost floor rather than a like-for-like price for equivalent answers.

He cited Airbnb’s reported pricing-policy classifier as an example of a workload where specialization and placement can be separated. The company described a fine-tuned 2.3B model reaching an F1 score of 0.902 on 10,000 held-out examples, with 35 ms median and 85 ms p99 latency on one RTX PRO 6000. Frontier APIs had similar reported quality but around 900 ms median latency. Osman used the case to argue that a small model trained on suitable examples can meet a specific task’s quality and latency requirements.

Whether to own or rent depends on the workload’s volume and shape. Osman pointed to a separate Airbnb comparison in which API pricing was cheaper below roughly 15–30 requests per minute, while one GPU became far cheaper at thousands of requests per minute. His own break-even illustration used a hypothetical $2-per-hour GPU, or $1,460 a month. At a listed output price of $20 per million tokens, it would need to be busy about 2% of the time to beat the API on price. At $0.60 per million tokens, the required utilization rose to about 72%. These calculations compare token prices, not task quality, and still require the local model to pass the relevant evaluation.

Osman also suggested renting for short jobs and bursts, potentially through a command-line workflow that starts a GPU and shuts it down when the work ends. He mentioned Modal’s CLI as one way to rent on demand, and suggested an agent could run the command and start the workflow. The point was not that rental removes engineering; it can make a short-lived workload economical if the system starts and stops the rented machine correctly. He acknowledged a threshold below which buying a GPU does not make sense.

There are reasons to use hosted services: renting avoids some maintenance and redundancy work, and operating a box means owning driver mismatches, broken dependencies, hardware failures and silent errors. Osman’s own render system supplied examples of these problems. He also warned that low-cost API listings may not make it clear which model or infrastructure serves a request. If using an endpoint, he argued, verify what it actually runs and whether it meets the required standard.

A deployable configuration should pin more than a model name. Osman’s checklist included the model and file revision, tokenizer and chat template, tool schema and parser, runtime and kernels, GPU, context, concurrency, precision and sampling settings. Acceptance tests should cover task cases and latency under load, with a fallback and rollback plan. When one part changes, rerun the checks that depend on it.

That discipline matters because the line keeps moving. Osman described open models improving in capability per parameter, task-specific models getting better at bounded jobs, and inference engines serving the same weights more efficiently. He cited Qwen’s reported results in which a Qwen 3.6 27B model matched or beat a Qwen 3.5 mixture-of-experts model with 397B total parameters on several coding benchmarks. His broader recommendation was to plan for the machine’s next workload as well as today’s, while rerunning evaluations with each release.

Continual learning would move memory into the model

Osman’s longer-term bet is that useful AI systems will learn from an individual’s outcomes, not just retrieve instructions from an ever-growing set of Markdown files. He argued that hosted providers train for a broad distribution of users, not for one person’s corrections and habits. Files and retrieval can help, but in his experience they can become bloated or conflict with one another. For a system to adapt its model specifically to a user, he believes, the relevant weights need to be under that user’s control.

He pointed to three research directions as pieces of a possible system. Engram memory stores information in a retrievable memory rather than changing the model’s backbone. Decision models return decisions rather than prose, making it possible to test thresholds. In-weights edits change specific weights in a smaller model. Osman’s proposed combination is a small model that can retrieve stored facts, make testable decisions and adapt its weights as it accumulates experience.

He separated the published pieces from the unbuilt system. The examples he cited showed a memory architecture, a sequence of edits to memory, and a repair to a small model; nobody, he said, had demonstrated those pieces working together as the continual-learning system he envisioned. That combination was his bet, not a result already established.

Learning also creates a new failure mode: a system can optimize for the evaluator rather than the work. Osman warned against rewarding only the final outcome. If every step shares one score, a lucky or faked finish may be rewarded along with correct work. Process checks can help, but their judges can also be wrong or gamed. The evaluator needs to remain outside the model’s control, and updates need a held-out test and a way back.

His proposed release loop was to log outcomes, curate and hold out examples, apply the shallowest useful change, gate the update on regressions and held-out cases, then promote it as a versioned, reversible release. If the gate fails—or the grader itself was within reach of the system—discard the update and keep the prior version. In this account, learning is not an untracked background effect. It is a deployment.

The operational choices remain useful before that longer-term bet pays off: place stages according to what each needs; size memory for context and concurrent users; distinguish fitting from serving; own steady, high-volume work and rent bursts when that is cheaper; keep data rules separate from model permissions; and rerun evaluations as models and runtimes change. Buy a GPU for a job it can do today, and build the system to take advantage of what becomes possible tomorrow.

The frontier, in your inbox tomorrow at 08:00.

Sign up free. Pick the industry Briefs you want. Tomorrow morning, they land. No credit card.

Sign up free