Longer Skills Files Shift Agent Design From Compression to Verification

Laurie Voss’s practical conclusion is that the constraint budget behind many agent designs has changed sharply. A year ago, engineers had reason to keep skills files below roughly 200 instructions, compressing rules and distributing them across sub-skills before models began losing track. In Voss’s test, current frontier models handled thousands of simple named constraints at once.

That removes a hard capacity constraint, but not the operational problem. The new bottleneck is verification: a fluent answer does not establish that the model followed every instruction. At high instruction density, models can forget, refuse, exhaust their reasoning budget, or return an apparently finished answer that quietly gives up halfway through.

The older 200-instruction figure was not folklore. Voss traced it to IFScale, a 2025 benchmark by Jaroslawicz and co-authors. Its method is deliberately narrow: ask a model to write a business report while requiring it to include a specified list of exact words—“customer,” “revenue,” and so on. The test varies the number of required terms, or density, and measures accuracy as the share that actually appears in the response.

Voss called the random-vocabulary requirement a proxy for rules about structure, tone, prohibited language, legal requirements, and conditional behavior. It is not proof that a long, realistic skills file will work. But it provides a favorable-case ceiling: if a model cannot retain 200 simple named constraints, it is unlikely to manage 200 more complicated ones.

Voss first reran the original result rather than assuming it still held. Of the ten models in the original paper, only three were still accessible through APIs: GPT-4.1, Claude Sonnet 4, and Gemini 2.5 Pro. His replication matched the original curves within their noise boundaries. In that older generation, accuracy began deteriorating around 200 to 300 constraints; by 500, models were losing roughly 30% to 50% of required words.

Voss then applied the same test to GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, and DeepSeek V4 Pro. At IFScale’s original maximum of 500 words, every model scored 100%. The benchmark had ceased to be difficult enough to reveal a limit, so he extended it from 500 to 1,000 terms, then 2,000, and ultimately to a 10,000-word vocabulary.

The new curves bent much further out. DeepSeek began dropping instructions around 750 rules and had lost nearly half by 2,000. Gemini 3.1 Pro remained strong through 5,000 instructions before a different failure mode emerged. GPT-5.5 held 99% accuracy through 5,000 rules. Voss’s broader estimate was that the new boundary lay closer to 2,000 constraints depending on the model, and reached 5,000 for the strongest results.

Voss cautioned against making the new result into another fixed design rule. The measurements were already aging as newer releases arrived, he said, and instruction capacity varied substantially by model. Engineering assumptions about how long a prompt can be should be retested rather than inherited from a model generation six or twelve months old.

The more consequential finding was not simply that the ceiling moved. The failure mode changed. Older models failed in the straightforward manner IFScale was built to measure: they omitted more required terms as density increased. The newer models could still fail at high densities, but their failures were model-specific and not always visible in a simple accuracy curve.

DeepSeek V4 Pro most closely resembled the old pattern. It simply forgot constraints, beginning around 750 rules and dropping nearly half by 2,000. Voss said this was the failure mode he trusted most—not because it is acceptable, but because it is predictable and easy to measure.

Claude Opus 4.7 produced a different kind of limit. IFScale’s random vocabularies sometimes included combinations such as “anthrax” and “cyanide,” which Voss said triggered Claude’s sensitive safety classifier. The model refused at the API level rather than completing the report. To run the test, Voss filtered word lists through OpenAI’s safety filter to remove terms that looked problematic; with that change, Claude performed well. His practical point was that a long skills file can encounter safety behavior before a model’s instruction capacity is exhausted, particularly around dual-use or medical material.

Gemini 3.1 Pro held up strongly through 5,000 instructions, then failed without forgetting them. Voss said it appeared to use thinking tokens to ensure it was following every constraint. At sufficiently high density, it consumed nearly its entire token budget thinking and left too little room to produce the requested report. The model could hold the constraints yet fail to provide an answer that used them.

GPT-5.5’s limit was subtler. When pushed far enough, it did not immediately refuse and did not merely omit terms. It began the requested report, then later explained that incorporating thousands of exact terms while maintaining a coherent business-report structure was impractical. Voss acknowledged that the artificial task was unreasonable—a coherent report on no particular topic containing thousands of random words—but it was still a benchmark failure because the unfinished response omitted most required terms.

The distinction matters most in production. Claude’s failure is loud: an API-level refusal makes it obvious that no result was delivered. GPT-5.5’s partial answer is harder to detect because it may look competent until its closing passage reveals that it abandoned the task. Receiving fluent text is therefore not evidence that the requested work was completed.

For the simple named constraints IFScale measures, Voss’s results weaken the old case for aggressively compressing every skills file. The previous playbook was to keep files under roughly 200 instructions and fan out to sub-skills, creating a layered structure to avoid overloading any one prompt. Current frontier models’ capacity for thousands of discrete constraints makes that degree of compression less necessary in many cases.

Where an application needs hundreds of straightforward requirements, Voss argued, they can now often sit together in one prompt rather than being split solely because of an assumed instruction ceiling. He described 2,000 named constraints as enough for an entire style guide: brand rules, formatting requirements, legal disclaimers, and other detailed conditions that might previously have been sharded across specialized agents. Consolidation can reduce the need to depend on clean handoffs among those agents.

The question has shifted from whether a model can hold all the rules to whether including them is worth the cost and latency. A prompt containing 10,000 distinct instructions may be technically feasible, but it will still be large, slow, and expensive. What used to be a hard wall is now a softer trade-off: whether the added instructions justify the cost and response time.

That is not an argument for eliminating structure indiscriminately. Voss’s result is about capacity for named constraints. The question of whether the model can reason through a complex task governed by those constraints is separate.

IFScale measures whether a model can track and reproduce named constraints. It does not measure whether a model can reason clearly over a giant prompt, resolve conflicts among rules, prioritize instructions, or apply them correctly to substantive work.

Voss pointed to Chroma’s 2025 “context rot” work across 18 models, which he said found accuracy on long inputs falling by 30% to 50% before models reached their context-window limits. He highlighted a counterintuitive result from that work: coherent, well-structured text was more likely to encounter this degradation than randomly shuffled instructions. Voss did not offer an explanation, but used the finding to delimit what his own benchmark establishes.

A model can therefore retain 2,000, 5,000, or perhaps more instructions without necessarily handling the tensions among them well. A long skills file may contain rules that conflict, conditions that require interpretation, and task material that has to be weighed against the instructions. Named-constraint capacity does not show that the model will make those judgments reliably.

Voss also cited a 2026 paper, “Revisiting the Reliability of Language Models in Instruction-Following,” which tested 46 models. As he summarized it, models that perform well on a benchmark can remain highly sensitive to superficial changes in wording or sequence. Rephrase an instruction, or place the same 2,000 instructions in a different order, and performance can change radically.

The question of what wording and ordering produce reliable compliance remains open, Voss said. He mentioned FireBench, CCR-Bench, and GuideBench as newer efforts aimed at the harder problem: measuring how well models follow many real, messy constraints at once. His own test was comparatively inexpensive—2,345 calls cost $209.19Voss reported 2,345 API calls at a total cost of $209.19.—but the conclusion is not that a cheap benchmark settles instruction-following reliability. It is that teams can afford to test concrete assumptions about their models rather than treating evaluation as prohibitively expensive research.

The practical consequence of these failure modes is that teams need to inspect whether requirements were met, not infer compliance from a model’s willingness to produce text.

An explicit refusal is easy to identify. Silent omission is not. Gemini may spend its budget thinking and return too little usable output; GPT-5.5 may produce a substantial, polished-looking report before quietly deciding not to finish it. In those cases, the model has not necessarily signaled failure in a way an application can safely rely on.

Laurie Voss’s recommendation is to treat model outputs as something to test, much as engineers test code. For difficult real-world tasks, he said, frontier models will encounter one or more of these failure patterns. Unless a model issues an obvious API-level refusal, detecting the problem requires monitoring the output—potentially with another LLM assessing whether the response met the relevant requirements.