Stable Prompt Prefixes Cut Long AI Agent Input Costs

A 50,000-token coding-agent session is not a 50,000-token bill. Alejandro AO explains that each new request typically resends the accumulated conversation—system instructions, user messages, tool calls, and tool results—along with whatever was just added. The model needs that full context to respond coherently, even when the new user request is small.

In his example, a session has reached 50,000 tokens and the user adds a 1,000-token question. The next model request contains 51,000 input tokens. If the model returns 3,000 tokens and the user follows up with another 1,000-token question, the subsequent request includes 55,000 tokens. The agent is not sending only the new 1,000-token increment; it is sending the transcript again, with a little more appended at the end.

APIs can obscure that mechanic. AO cites the Responses API as an interface that can make the model appear to remember prior turns, while the underlying system still reprocesses the conversation context for each response. That is why a session that eventually contains 50,000 tokens can have incurred input processing for 50,000 tokens, then 51,000, then 54,000, then 55,000, and onward.

The cost problem compounds because each turn adds to the transcript that will be sent again on later turns. AO’s concern is not merely that a context window can become large; it is that the growing history is repeatedly processed at each step. Without a discount for the repeated part, long coding sessions can become expensive quickly.

The familiar database-cache analogy is misleading here. In a conventional response cache, an application repeats a query and receives a stored result without asking the database to do the work again. Applied to an LLM, that would imply storing a prompt-and-answer pair and returning the old answer when the same prompt arrives.

AO calls that the wrong model for agent work. Caching an LLM output might technically be possible, but it is not useful when the agent is continuing a conversation and needs to produce a new response. What prompt caching reuses is the input the model has already processed: the stable beginning of a new request, including system instructions and earlier conversation history.

A request can therefore contain both discounted and fresh input. If the beginning exactly matches a previously processed prefix, that repeated portion can be charged as cached input. The new material appended at the end still needs fresh processing. AO describes providers as routing repeat processing to the same machine that handled it before, allowing the request to hit a warm cache.

That makes exact prompt structure economically important. The useful pattern is a stable system prompt followed by an append-only history. If an early part of that sequence changes, the later request no longer has the same reusable prefix—even if most of the text remains similar.

AO frames prompt caching as a large input-price differential. The first processing of a token is charged at the ordinary input rate, and some providers charge more than that rate to write input into cache. Subsequent reads of the same input can be much cheaper.

The comparison shown on screen lists cache-hit prices from 0.008 times to 0.15 times the base input price across the providers and models displayed. OpenAI and Anthropic are each shown at 0.1 times the base rate for cache hits; DeepSeek V4 Pro is shown at 0.008 times.

AO characterizes the input price of GPT-5.6 Sol and Claude Opus 5 as roughly $4 per million tokens. In his simulated 200,000-token coding-agent session, the uncached cost for both models reaches about $41. He describes 200,000 tokens as a short session, underscoring that repeated full-price reads can become costly before an agent’s context appears exceptionally large.

The displayed comparison covers Claude Opus 5, GPT-5.6 Sol, Gemini 3.1 Pro, Kimi K3, Grok 4.5, and DeepSeek V4 Pro. In each case, the cached bar is substantially lower than the uncached bar in AO’s scenario. He particularly calls out DeepSeek as unusually inexpensive in that comparison.

The accompanying cumulative-cost chart supplies the underlying logic. Without caching, the rising transcript is billed again at the ordinary input rate on later turns, so the session-cost line climbs sharply as context accumulates. With caching, most of the old transcript is charged at the cache-hit rate while only the appended material is fresh, leaving the cost curve relatively linear in the example.

Whether that happens depends on provider behavior as well as harness design. AO says caching is commonly handled behind the scenes in Responses-style APIs, but should not be assumed when using a chat-completions API. He says OpenAI and Hugging Face Inference Providers cache input automatically, while Anthropic and Gemini require the agent making the API calls to enable the relevant caching behavior.

A cache-friendly harness protects the part of the request that can be reused. AO identifies the system prompt and conversation history as the critical inputs: keep the system prompt stable where possible, and preserve history as an append-only sequence rather than changing earlier material.

The system prompt matters disproportionately because it appears at the beginning of every request. AO uses a hypothetical 10,000-token system prompt in a conversation that grows beyond 200,000 tokens. If that prompt remains unchanged, it can remain part of the reusable prefix throughout the session. But a timestamp, changing current working directory, or dynamically updated tool list changes that early input and invalidates the cache after the changed material.

The point is not that an agent can never use dynamic state. AO’s warning is specifically about placing dynamic values in the system prompt, where a small change disrupts the prefix on which every later request depends. His recommendation is to avoid putting timestamps, current working directories, and dynamically updated tool lists there when preserving cache hits matters.

Compaction creates a different, intentional reset. When a harness replaces a long history with a shorter summary, it has transformed the prior context into a new prompt. AO says that invalidates the existing cache: the compacted prompt must first be processed as fresh input, after which its own repeated prefix can begin receiving cache-hit pricing. Compaction is therefore not a mistake, but it is a cost event to account for.

Time is another source of resets. Cache entries expire after provider-specific intervals. AO says that on Hugging Face Inference Providers, the duration may depend on the provider to which the request is routed. He gives one hour of cache on OpenAI’s OAuth API as an example, and says Anthropic defaults to five minutes through its API, with one hour in the Claude Code-authenticated case.

For an operator, the distinctions are concise:

• A low hit rate at the start of a session is a normal cold start: no reusable prefix has yet been established.
• A drop after an extended pause can reflect cache expiry.
• A drop after editing the system prompt or rewriting history reflects prefix invalidation.
• A reset after compaction reflects a newly summarized context replacing the earlier transcript.
• A provider that does not cache automatically needs its caching behavior enabled in the agent’s API calls.

AO recommends exposing cache behavior in the harness rather than treating it as invisible billing machinery. Pi’s status line displays CH for cache-hit rate; the Kimi K3 example shown reads CH64.5%. Tau displays total input and output tokens, estimated cost, the latest-request cache-hit rate, and the session-wide rate.

The Tau session shown is an illustration of what those measurements can reveal. Its sidebar reports 10.9 million input tokens and 45.6 thousand output tokens, with an estimated cost of about $0.06. The latest request has a 100% cache-hit rate and the displayed session rate is 97%.

The exported Tau dashboard gives a more precise aggregate view: a 97.1% cache-hit rate, 10,536,263 cached input tokens, and 320,002 fresh input tokens. Those figures are displayed for AO’s particular DeepSeek V4 Flash session through Hugging Face Inference Providers, not as a general cost guarantee.

AO also shows a request-by-request cache report. He identifies the blue series as cache hits and the red series as fresh input or cache writes, while noting that cache writes are not correctly measured in that display. The pattern begins with fresh tokens, then shifts toward cached rereads as the unchanged conversation grows.

In the displayed session, two visible resets correspond to breaks for lunch and dinner. AO says the cache expired during those pauses. The next request had to write fresh context again; later rereads returned to discounted cache-hit pricing. That is the practical value of monitoring: a harness can reveal whether a cost increase came from a normal cold start, an idle-period expiry, compaction, or a change to the prompt prefix.