Long Context Cannot Replace Persistent Learned Memory
Engram co-founder and CEO Dan Biderman argues that larger context windows and retrieval systems cannot give AI the accumulated understanding needed to work reliably across an organization’s expanding knowledge base. Models that must reread documents, histories and decisions on every query are costly and prone to “context rot,” he says; Engram’s answer is a trainable memory layer that compresses recurring knowledge into learned representations. The longer-term challenge is deciding what a model should internalize, keep in explicit records and update over time without degrading performance.
The constraint is not context length alone, but repeatedly starting cold
? dan-biderman says AI systems working over organizational knowledge will eventually run into a limit that larger context windows alone cannot solve. A frontier model can be given more documents, a larger retrieval set, or a longer-running transcript. But if it has to reconstruct a company’s world from scratch on every query, the result is both expensive and, he believes, increasingly unreliable.
The immediate problem is repeated reading. Documents, client histories, code, decisions, policies, and the output of other agents may all be processed again and again by models that have no persistent understanding of the organization. Biderman expects proprietary corpora to grow sharply as agents produce more artifacts. Within 18 months, he says, some AI-native companies could plausibly have internal data measured in trillions of tokens—“internet-scale data” inside a single organization.
Text-based knowledge systems will still matter at that scale. A company can maintain wikis, indexes, retrieval systems, and explicit records that users can inspect. But those systems must remain current, and the models using them still need to process their contents. His concern is that text cannot be the only representation of an organization whose knowledge base is expanding continuously.
The second constraint is quality. A model may accept a very large context without returning an error, Biderman says, but that does not mean it can reason holistically across it. More context can introduce distraction and reduce accuracy, a failure mode he calls “context rot.” He expects that problem to remain even as context windows reach millions of tokens.
The current LLMs are like coming into the kitchen, first time every time, reading the textbook, cooking the dish, measuring everything, but they don’t have the intuition of a chef.
The analogy distinguishes repeated access to instructions from accumulated operational judgment. Current models can read recipes and reproduce procedures; what they lack, in Biderman’s telling, is the learned sense of how dough should feel, how much salt a dish needs, what has worked before, and how to improvise beyond a written procedure.
His claim is not that notes, documents, or retrieval are unnecessary. Persistent learned state should reduce the need to reconstruct a familiar world from raw text each time a task begins.
Cartridges turn a corpus into a learned representation
Engram’s proposed mechanism is a set of compact, trainable representations it calls “cartridges.” A cartridge can represent a corpus, a task, or a domain. Instead of supplying all relevant information as raw context at inference time, the system studies that material beforehand and loads a learned representation when it needs to answer questions about it.
? dan-biderman characterizes the process as closer to pretraining than retrieval. Given a large corpus, a model can ask itself questions, generate quizzes, attempt problems, and use gradient descent to build a compact representation of what it has learned. He calls cartridges “capsules of knowledge” that can be loaded into and out of a model as a state describing its world.
The displayed Cartridges paper describes an offline-trained KV cache for a specific corpus. Rather than training that cache only on next-token prediction over the corpus, its self-study method generates synthetic conversations about the material and trains with a context-distillation objective. Its abstract reports that self-study-trained cartridges matched in-context-learning performance on its challenging long-context benchmarks while using 38.6 times less memory and enabling 32.8 times higher throughput.
The displayed Minions paper addresses a related problem: how an on-device model with local access to long documents can work with a cloud frontier model without repeatedly sending or processing the full context remotely. Its simpler local-remote baseline reduced remote tokens by 30.4 times, but recovered only 87% of the frontier model’s performance. The paper attributes that shortfall to the local model’s difficulty following extraction instructions and reasoning over long context. Minions instead has the remote model decompose a task into easier, shorter subtasks that the local model executes in parallel.
| Project shown | Approach | Reported result |
|---|---|---|
| Cartridges | Offline-trained KV-cache representation using self-study | Matched in-context-learning performance on stated long-context benchmarks with 38.6× less memory and 32.8× higher throughput |
| Minions | Frontier model decomposes tasks into shorter local subtasks run in parallel | 5.5× lower average cost while recovering 97.4% of remote frontier-model performance |
| Simple local-remote baseline | Local model reads full context and communicates with a remote model | 30.4× reduction in remote tokens while recovering 87% of frontier-model performance |
Biderman characterizes the larger opportunity as potentially “a thousand X more compressed” than raw textual context, but not as a universal benchmark. The point is representational: train once on knowledge that recurs across many queries, then reuse what was learned rather than treating every query as a first encounter.
That extends a theme from his earlier work in semi-supervised learning. His academic work focused on useful generalization from limited examples and limited resources. At Engram, that efficiency question moves from training data to inference: how much context, memory, and compute a system needs in order to perform a task.
Some enterprise questions cannot be retrieved from one place
? allen-park presses the obvious alternative: if retrieval can find the useful passages from a cookbook, why not simply provide those passages to a strong model? ? dan-biderman does not treat retrieval-augmented generation as obsolete. “We do not take the bet that no notes need to be taken,” he says. Textual systems remain valuable because they are interpretable, traceable, and often exactly what a model needs.
The limitation appears in questions for which no single passage contains the answer.
Biderman gives an example from Engram’s work with Harvey. A law firm or investment bank may have many client matters containing documents related to financings, loans, mergers, and acquisitions. Consider the question: which M&A deals have not been completed this year?
The answer may not appear in a document stating that a deal failed to close. To answer it, a system may need to work through client matter by client matter, interpret the status of each, determine whether the loop was closed, and aggregate those judgments across the firm. Biderman calls these “ambient, hard questions”: problems where one can get the gist by reading everything but cannot identify the answer through one obvious retrieval target.
A frontier model can attempt that process through long context, compaction, and agentic loops. But Biderman says such a query can consume thousands of dollars even though an employee familiar with the organization might consider it routine. The case for learned memory is not merely cheaper search. It is the prospect of associations across a corpus that can support questions whose answer is distributed across many files.
This is why he reaches for web-scale pretraining as an analogy. Models are trained on broad data and can subsequently generalize, interpolate, and extrapolate; they are not generally operated by retrieving a few passages from the entire web for every task. Engram’s enterprise bet is that a company’s proprietary world can receive a narrower version of that treatment while remaining connected to explicit records and auditable context-management systems.
Memory allocation is an unsolved problem, not a fixed policy
? dan-biderman does not claim that every fact should be embedded in weights. He calls the division between internalized and externalized knowledge “a major open question,” both for AI and for the study of human memory.
Compaction—where a model manages its context by retaining some tokens and evicting others—is improving quickly, he says, and will be part of the solution. Yet it is lossy by definition. Deep into a long session, current systems can become confused or forgetful, in his view. A learned memory trace would also be lossy, but would preserve information as a distributed representation in weights rather than as a smaller textual record.
Perfect retention would not necessarily be desirable. Biderman notes that people who remember everything they have experienced are not necessarily helped by it; too much retained material can be distracting or frightening. The question is which information deserves durable representation and which should remain available in notes or other external systems.
He offers a rough intuition rather than a rule: a hotel room number for one night is unlike an address or a partner’s phone number. Salience, recurrence, and what a fact enables the system to do may all matter. But a manually maintained taxonomy—this belongs in weights, that stays in text—would become “whack-a-mole” because people and enterprises differ too much in what matters.
The longer-term objective is for the model itself to learn the allocation: when to consult a notebook, when to rely on its learned state, and which new information should change either representation. That means training models to manage memory without explicit supervision for every individual decision.
A personal model would make feedback immediately consequential
? dan-biderman puts enterprise knowledge first because it is concentrated, dense with expertise, and already a major site of AI use. Parameter-efficient techniques such as LoRA, cartridges, and memory layers could be applied to those corpora while operating alongside records and context-management methods that people can inspect and audit.
But the more ambitious version is personal. Biderman imagines every person having a model, part of a model, or a set of weights representing their knowledge and expertise. The more time they spend with it, and the more useful information it receives, the more capable it becomes for that particular person. He compares the relationship to a Tamagotchi: something a user nurtures and improves over time.
That framing changes the meaning of feedback. A thumbs-up or thumbs-down on a general-purpose AI product may or may not affect a later release. In Biderman’s conception, feedback about a personal model would initiate compute devoted to making that user’s system better. The user would not simply be contributing data that might improve a provider’s general model; they would be shaping a model intended to improve specifically for them.
The system still cannot treat every user statement as ground truth. Users can be wrong, he notes, and models will increasingly know things their users do not. The model needs to distinguish feedback worth incorporating from feedback that should be discounted or ignored. The goal is not to put a person in charge of every memory decision, but to build a tighter loop in which user feedback can inform training while the model learns how to evaluate it.
Biderman expects personal, continually updated learned states eventually to run on personal devices as hardware improves. That is a longer-term aspiration. Enterprise deployments are the nearer proving ground.
Replacing prefill creates a serving problem at a different scale
? dan-biderman frames the same issue as a systems problem. Using a Llama 70B model as an illustration, he says a single Wikipedia article may be only tens of kilobytes as text while the model state associated with reading it can occupy roughly 80 gigabytes in GPU high-bandwidth memory. He compares that with an FP16 parameter set of roughly 140 gigabytes, which represents knowledge from vastly more data.
The comparison is meant to show how memory-intensive raw-context serving can be. Feeding a large corpus into a transformer creates a KV-cache burden that grows with the input. Chip design, kernels, and serving software can reduce that burden, but Biderman says Engram is trying to move part of the work outside query-time prefill altogether.
Rather than having a model reread a corpus before each answer, the proposed approach spends training compute earlier, loads a learned representation later, and begins decoding immediately or after only a small prefill. Biderman jokingly calls this “destroying prefill.” He places the idea alongside the trend toward separating prefill and decode work onto different specialized hardware.
The infrastructure requirement grows sharply if the personal-memory vision works. Instead of serving only a shared base model and a common retrieval index, a system may need parameter-efficient adapters or learned states for millions of people and teams. Those states would need to be stored, updated, moved from disk into GPU high-bandwidth memory, swapped at inference time, and coordinated with other workloads.
That is why continual learning is not only a model-training problem in Biderman’s account. Researchers must determine how to update models without damaging them and what data they should learn from. Infrastructure engineers must make a large fleet of distinct, changing learned states practical to serve. He specifically points to a need for LLM performance and research engineers as well as engineers able to build the APIs, databases, and systems for moving those states through a large compute footprint.
Efficiency changes which tasks are feasible
? dan-biderman treats token efficiency as a capability question, not a discount feature. It is easy, he says, to think of efficiency as saving dollars while intelligence comes from spending more compute. But resource use determines which tasks a system can realistically attempt.
The more you can do with less, the more ambitious tasks you can solve longer term.
The existing scaling paradigm has been “doing more with more,” he says. He does not dismiss that path; it has produced major advances and will remain valuable. But he expects another dimension of progress to come from doing more with less: using fewer tokens, less memory, and less repeated work to handle longer-horizon and more adaptive tasks.
That logic also shapes his view of model routing. Many requests do not require the most capable or costly model; others do. He expects the eventual stack to be multi-model rather than a system in which Engram’s learned memory eliminates the need for frontier models.
A memory layer could act as a persistent colleague that has been “looking over your shoulder,” knows where to focus, and can send a more targeted problem to a powerful system when the task warrants it. Routing remains difficult: models, versions, and prices change quickly, and choosing the right system at the right time and cost is still unresolved.
Engram’s current focus is narrower than the eventual vision: helping models reason over large corpora with fewer tokens and less confusion. Beyond that, Biderman expects gradient-based updates during extended tasks—what some call test-time training or test-time compute—to matter for difficult work in science, engineering, and defense.
He is explicit that these are open problems. Continual learning without degrading a model, deciding what to internalize, evaluating feedback, and serving large numbers of learned states all require further breakthroughs. The commercial requirement is more immediate: the team must build products customers want, not operate only as a research group.