Agent Memory Requires Extraction, Separate Stores, and Hybrid Retrieval

Alejandro AO draws a hard boundary between conversational history and long-term memory. An LLM is stateless: it processes a prompt and produces a completion, but a later prompt carries no inherent awareness of the previous exchange. Agent scaffolds compensate within a session by appending user messages and model responses to an active history, then resending that history on subsequent turns.

That is conversational memory. It lets an agent refer back to something said earlier in the same session, but it does not survive a new conversation. Long-term memory is an external system that persists independently of the session store, records information from many conversations, and retrieves relevant material when a later turn needs it.

The engineering proposition is therefore broader than “put chat logs in a vector database.” A persistent memory layer needs to decide what should become a durable memory, preserve enough context to interpret new statements, store information in forms that can later be searched, and rank retrieved material before it enters an agent’s context.

A user-centered design can also make memory portable across agents. A person’s preferences, current projects, and other retained context could be shared among different assistants rather than remaining inside one agent’s conversation history. The same architecture can separately hold information about the user, information learned by an agent, and information associated with a particular run.

The system can expose memory as an explicit agent tool—something like a search memory database call—or retrieve automatically on each turn. In the automatic case, the user’s message becomes a search query, relevant memories are added to the context, and the LLM receives the augmented prompt. In the explicit case, the agent decides when and how to search.

Mem0’s ingestion flow can run after every agent turn, taking the messages produced during that turn and converting them into stored memory. Alejandro describes three modes, which imply materially different answers to the question of what the system should retain.

Procedural memory summarizes the process an agent followed: its actions, tool calls, and results. The purpose is to let the system later retrieve and reproduce a procedure. Alejandro says this can be useful after an agent completes a task whose method matters, but he does not think it is used much anymore; extracting a reusable skill from a conversation transcript, in his view, works better.

A simpler mode sets infer to false. The pipeline embeds the incoming messages and stores those messages directly. No model determines whether a statement is a preference, a durable fact, or an incidental detail.

The more selective mode sets infer to true. Here, messages are passed to an LLM acting as a memory extractor. Its output is structured JSON containing the memories it believes should be saved. This is the mode Alejandro treats as the more sophisticated long-term-memory approach.

The extraction model does not receive only the latest message. Its prompt includes:

• its role as a memory extractor;
• a summary of the user;
• the incoming messages;
• recently saved memories;
• memories retrieved as relevant to the new messages;
• the last 10 messages sent through the pipeline;
• the conversation date and current date.

That context determines whether extraction can make sense of a statement. If a user says, “He is really good at this,” the statement is not useful as durable memory unless the extractor can resolve who “he” is and what “this” refers to. Mem0 uses the retained recent-message window for that short-range grounding.

The relevant-memory portion of the prompt is itself retrieved from the existing main vector store. Mem0 flattens the incoming messages into a string, embeds it, and searches for related stored memories. Those results help the extractor interpret a new statement in light of prior knowledge.

Once the extractor returns a candidate memory—such as “The user prefers vegan restaurants”—the system writes it with metadata, creates a hash for deduplication, and saves a lemmatized version for later lexical search. The memory system is therefore not simply preserving dialogue: it is producing compact claims about the user or agent and placing them into a retrieval-oriented data model.

Alejandro calls LLM-based extraction the most important part of the ingestion flow. He says, with the qualification that he may be mistaken, that Mem0 uses GPT-4o-mini for this step. His larger implementation point is that extraction is straightforward enough to run locally with a small open model.

Alejandro AO describes Mem0 as relying on three stores with distinct jobs: a main memory vector database, an entity-memory vector database, and SQLite.

The main store contains the memory itself, generally a short sentence or paragraph, plus metadata. For a statement such as “The user prefers vegan restaurants,” that metadata can include whether the memory concerns the user or the agent, creation and update dates, an expiration date, the agent that created it, a hash, and a lemmatized form of the text.

The hash supports deduplication, but only at the level of exact wording. Alejandro flags the limitation directly: two memories with the same meaning but different phrasing will not be treated as duplicates by this mechanism. It is a lightweight check against repeats, not a semantic-resolution system.

Lemmatization supports the later keyword-ranking stage. Inflected forms are reduced to a shared dictionary form, allowing lexical matching to recognize related variants such as “running” and “ran” without requiring identical surface wording.

The second vector store holds entities rather than full memories: people, proper names, places, and similar references extracted from the main-memory records. Each entity links back to one or more main memories. If the system stores that a user’s favorite neighborhood in Paris is Montmartre, “Paris” and “Montmartre” can become entities associated with that memory. A later question about Paris can retrieve the entity and identify linked memories that may deserve more weight.

SQLite serves a different purpose. It logs changes to the vector stores and retains the 10 most recent messages that passed through the pipeline. Those messages are not the long-term memory layer; they provide enough immediate context for the extraction process to resolve references and interpret a new message correctly.

The design separates three different kinds of state that are easy to collapse into one database: durable memory claims, structured links among named concepts, and the local conversational context needed to interpret fresh input.

Mem0’s retrieval path starts with a query, a requested top-k, a minimum-score threshold, and an identity scope: whether the search concerns user memory, agent memory, or memory tied to a run. The scope matters because user memories may concern preferences and prior experience, while agent memories can concern learned facts or procedures.

The query is embedded with the same embedding model used to construct the main vector database. Mem0 then runs approximate-nearest-neighbor search to produce a semantically similar candidate pool. But that initial vector result is not the final ranking.

If the application asks for 10 memories, for example, the vector search retrieves 60 candidates. Alejandro’s explanation is that reranking works better over a larger pool. Crucially, the later ranking stages cannot introduce memories absent from that semantic pool; they only change the ranking of what vector search already returned.

The first additional signal is lexical matching with BM25. Mem0 lemmatizes the query, compares it with the lemmatized memory text, and produces a keyword-overlap score from 0 to 1. This gives the ranking a way to reward direct term overlap that semantic embeddings may not emphasize.

The second signal is entity boosting. The system extracts entities from the query, searches the entity vector store, and identifies main memories linked to the resulting entities. A memory gets this boost only if it was already part of the initial semantic candidate pool.

The boost is designed to favor specificity. A broad entity such as Paris, linked to many memories, is less discriminating than a specific entity such as Montmartre, linked to only a few. The displayed weighting reduces the boost as an entity’s number of linked memories grows: n_linked is at least one; the memory weight is 1 / (1 + 0.001(n_linked - 1)^2); and the boost is entity similarity multiplied by the entity and memory weights.

Alejandro’s practical interpretation is straightforward: an entity linked to fewer memories is more likely to identify a narrowly relevant memory, so its associated records receive a stronger boost. The entity contribution ranges up to 0.5; vector similarity and BM25 each range from 0 to 1.

For each candidate memory, Mem0 adds the semantic, keyword, and entity scores. The maximum possible total is 2.5, so the sum is divided by 2.5 to yield a final score. The system then applies its threshold and returns the requested top-k memories.

That constraint defines the architecture’s trade-off. Entity links and lexical matching can correct the order of semantically plausible results, but they cannot rescue a relevant memory that the original vector search failed to retrieve. The vector index remains the gatekeeper; the hybrid signals refine its decisions.

Alejandro AO recommends a small text-generation model for extraction because the task is to produce structured memory statements from supplied context, not to generate long-form answers or execute an open-ended task. He places an appropriate range at roughly 1 billion to 12 billion parameters, says he would not go below 1 billion unless the model is fine-tuned, and identifies Qwen-3-8B as a plausible option within that range.

For retrieval embeddings, he lists several models intended to run locally:

• BAAI/bge-m3 for multilingual dense and sparse retrieval;
• intfloat/e5-large-v2 for English retrieval;
• nomic-ai/nomic-embed-text-v1.5 for long-context embeddings;
• gte-Qwen2-1.5B-instruct for instruction-aware retrieval.

Alejandro’s implementation requirement is to use the same embedding system for the query as for construction of the vector database. The retrieval comparison depends on that shared embedding setup.

He points to Hugging Face’s feature-extraction listings and the MTEB leaderboard as ways to compare embedding models by language and domain, including medical, legal, and code-related benchmarks. For a system intended to run over the long term, he recommends fine-tuning smaller models for the application’s own extraction and retrieval needs.

Alejandro also identifies an improvement that sits outside the Mem0 flow he describes: query rewriting. The raw user message is usually the retrieval query, and he says Mem0 does not, to his knowledge, rewrite queries by default. He recommends implementing that upstream in the agent harness, using a small LLM to turn an indirect or conversational user request into a clearer search query.

That matters because query rewriting changes the input to the semantic candidate generator—the stage that determines which memories are even eligible for reranking. Better entity boosts and BM25 scores can improve the ordering of the candidate pool, but they cannot compensate for a query that fails to retrieve the relevant pool in the first place.