AI Agents Depend on Retrieval, Tool Use, and a System Harness

Coding tools have moved from returning a code snippet to taking on a task: changing files, writing and running tests, and potentially deploying or launching software. The lecture uses that progression—from copy-and-paste prompting, to IDE autocomplete, to agentic coding—to frame its central question: what had to be added around a language model for it to do more than produce a plausible next response?

Two limitations provide the starting point. First, a model’s knowledge is bounded by its training data. An off-the-shelf model cannot know the contents of a private codebase it has not seen, and its stated knowledge cutoff marks the limit of information it can rely on from training. Second, a model that can answer a request does not thereby perform the action the request describes. Asked to order a teddy bear, a plain model can respond in text; it cannot place the order by itself.

Adding all potentially useful information to every prompt is not a general solution. Context windows are finite, and a large window does not guarantee strong retrieval from everything placed inside it. The lecture uses a “needle in a haystack” test to illustrate the concern: a fact is inserted at different positions in inputs of different lengths, and the model is asked to find it. The displayed results show retrieval degrading at larger context lengths, with performance varying according to where the fact appears. More context also means more input tokens to pay for. The practical goal, then, is not simply to give a model as much information as possible. It is to make the relevant information available when needed, and to let the system take actions when an answer alone is insufficient.

The lecture builds that system in layers. ? afshine-amidi develops the retrieval account: how to build a knowledge base, find likely relevant material, rank it, and evaluate the result. Shervine Amidi develops tool use and the surrounding machinery: how models invoke functions, how systems choose which tools to expose, and how agents manage context, memory, and work in an execution environment.

Retrieval-augmented generation, or RAG, puts relevant information into a prompt so that a model can answer a query using material beyond its training data. Its three steps are retrieve, augment, and generate: search a knowledge base for relevant documents, add the selected material to the prompt, and generate a response from the augmented prompt.

The retrieval step is decisive. The model can only make use of material that the system selects and supplies; sending the wrong passages, or missing the right ones, undermines the rest of the process. The lecture therefore treats RAG not as a single operation but as a pipeline with a knowledge base, a candidate search, a more precise ranking stage, and an evaluation method.

Building the knowledge base starts with collecting documents and dividing them into chunks. Each chunk is represented by an embedding vector, which allows the system to compare its meaning with a query. The choices involved include embedding size, chunk size, and overlap between chunks. The lecture describes embeddings as typically having dimensions in the thousands, chunks as typically being hundreds of tokens, and overlap as often around 10 percent. These are presented as useful orders of magnitude, not fixed rules.

Chunking creates a trade-off. Larger chunks carry more context but risk making the model’s input too large. Small chunks are easier to retrieve and fit into a prompt, but can separate a passage from the context that makes it intelligible. Overlap reduces the chance that a boundary cuts off useful information or splits a sentence from the material needed to understand it. Even with overlap, a chunk may not make sense on its own. One proposed improvement is to add a short contextual description derived from the whole document, so that the chunk is more self-contained when indexed and searched.

That contextualization step can itself consume model calls and tokens. The lecture describes putting the whole document at the beginning of a prompt and asking the model to situate a particular chunk within it. Keeping the document in a stable prompt position can make that repeated input eligible for caching; the chunk-specific portion changes from request to request. The point is not that contextualization is free, but that prompt structure can affect its cost.

Retrieval itself is usually split into two stages. Candidate retrieval scans a potentially large knowledge base and aims to maximize recall: bring back a broad set that is likely to contain relevant material. Ranking, or re-ranking, then gives more careful relevance scores to the smaller candidate set and aims to maximize precision. The system may select, for example, a hundred candidates first and then identify the top K passages to place in the prompt.

For the first stage, a bi-encoder represents the query and each chunk separately. The chunk embeddings can be computed in advance, leaving the system to embed the incoming query and compare it with stored vectors. Approximate nearest-neighbor methods can make that search practical across large collections. The knowledge base may change as documents are added, but query and chunk vectors must remain comparable; the lecture notes that the same embedding model should be used.

Semantic similarity is useful but can retrieve material that resembles a query without answering it. In the lecture’s example, the query is “Where is Cuddly?”—Cuddly being a teddy bear—and candidate passages include “Huggy likes to work downstairs,” “he was in a cuddly mood,” and “Cuddly spends most days surrounded by books.” An embedding-based search might favor “Huggy” because of its association with “Cuddly,” or another passage because it resembles the query in form. The passage that actually answers the question may not be the one that scores highest.

Keyword retrieval can correct some of these misses. BM25 scores documents by the exact words shared with a query; for “Where is Cuddly?”, it could retrieve passages containing “cuddly” or “Cuddly.” But keyword matching also fails when the same idea is expressed in different words. In practice, the lecture says, systems often combine semantic and keyword ranks in a hybrid search rather than relying exclusively on either.

Other techniques address mismatches between queries and documents. A question and a declarative passage have different forms, so their embeddings may not align well. Hypothetical Document Embeddings (HyDE) addresses this by generating a possible document in response to the query and comparing that generated text with the indexed chunks. A query that contains several questions can also be rewritten into separate questions, each used for retrieval. These methods are examples of ways to improve the first pass, rather than replacements for the overall retrieve-and-rank pipeline.

The second stage uses a cross-encoder. Instead of encoding the query and passage separately and comparing their vectors, it feeds the query and a candidate passage together through one encoder. This allows tokens in the query and document to interact through attention before the model assigns a relevance score. The lecture’s example is a request for a teddy bear with a red scarf and blue socks. A bi-encoder might represent that description similarly to one involving a blue scarf and red socks; a cross-encoder can model the relationships between the color terms and the items they describe.

A cross-encoder is more expressive, but it is too costly to apply across an entire database. It is practical after candidate retrieval has reduced the number of passages to score. The lecture describes an encoder-only transformer as one possible architecture: the combined query and passage pass through a bidirectional encoder, and a representation such as the CLS token can be used to predict whether the pair is relevant. Training requires examples of relevant and irrelevant query–passage pairs.

The distinction between the stages also explains where vector databases fit. They can store chunks and their embeddings for candidate retrieval. They do not eliminate the need for a re-ranker: a joint query-and-chunk score cannot be precomputed in the same way, because it depends on the particular query.

A retrieval system’s output is an ordered list, but a high score is useful only if it corresponds to the goal of the task. The lecture describes evaluating a system with labeled queries: for each query, a dataset identifies which documents are relevant, the system returns a ranked list, and its predictions are compared with those labels.

The measures emphasize different qualities. Recall at K asks what fraction of all relevant documents appeared among the top K results. Precision at K asks what fraction of the top K results are relevant. Candidate retrieval tends to prioritize recall so that relevant material is not missed; the final ranking aims to improve precision so that the passages supplied to the model are useful.

The distinction matters operationally. If the first stage misses a relevant passage, re-ranking cannot recover it: the passage is no longer among the candidates it can score. That is why candidate retrieval is designed to cast a relatively wide net. But an overly broad candidate set can fill the prompt with irrelevant material, so the ranking stage must sort useful passages upward and keep noise out of the final context. Recall and precision therefore describe different failure modes rather than competing names for the same goal.

Reciprocal Rank focuses on the position of the first relevant result: it is one divided by that result’s rank. It rewards a system for putting at least one useful passage near the top. It is especially sensitive to the first relevant result; other relevant passages lower in the list do not change that first-result position. NDCG, by contrast, takes the ordering of multiple relevant items into account.

Normalized Discounted Cumulative Gain at K, or NDCG@K, considers relevance throughout the top K and discounts lower-ranked items. In the binary formulation shown in the lecture, each position contributes its relevance divided by a logarithmic discount based on its rank. The result is divided by the score of a perfect ranking, giving a value between zero and one; a value nearer one indicates a better ordering. The discount captures the intuition that a relevant passage near the top is more useful than one buried lower down. Normalization makes scores more comparable across queries with different numbers of relevant passages by comparing each ranking with its own ideal ordering.

Recall and precision answer still different questions. A system can have high recall at K by finding most of the relevant material, while also having low precision if many irrelevant items occupy those K places. It can have high precision if most of its top results are relevant, yet low recall if it misses many other relevant items. Which balance matters depends on the stage and the application: the lecture associates broad candidate retrieval with recall and the final ranking with precision.

These metrics do not make relevance objective in every application. They make performance measurable against the labels and ranking goals chosen for a task. They also clarify why “retrieve the top K” is not a complete specification: a system might retrieve many relevant passages but rank them poorly, or return a clean top few while missing other relevant material. The lecture notes that implementations vary in how many retrieved passages they put in the prompt and whether they apply a score threshold. A common approach is to include the selected K passages, often ordered from most to least relevant.

Code search is a useful exception to the natural-language retrieval pattern. A coding agent often needs a particular function or symbol, and semantically similar passages may not be as useful as exact occurrences. The lecture points to grep—“global regular expression print”—as a way to find every location where a search term appears in a codebase. In that setting, exact matches can be more predictive of what the user wants than semantic similarity. The broader lesson is that retrieval method should follow the shape of the task: similarity and keyword combinations suit many natural-language searches, while code tasks often benefit from direct search over terms and patterns.

RAG can help answer “Where is Cuddly?” if a knowledge base contains an up-to-date passage saying where Cuddly spends time. It is not enough for “Find a bear near me.” That request depends on changing information, such as current inventory, and on the user’s position. A static passage may be outdated, and the model does not automatically know where the user is.

Tools address this gap by connecting the model to functions that can perform actions or query external systems. In the teddy-bear example, one function retrieves the user’s location; another searches a backend for a bear near those coordinates. The flow has three stages: the model selects a function and supplies arguments, the system executes the function and receives a result, and the model interprets that result into a response. The middle stage is ordinary backend execution, not something the language model performs by generating text alone.

A function’s implementation may be written in Python, but the model does not need to see all of that code. It needs an API description, the expected inputs, and the output format. Clear documentation matters because the model must choose the right function and provide usable arguments. Inputs should be explicit and typed—for example, a location argument expressed as a latitude–longitude pair. Outputs should also be structured and interpretable, so that the model can turn the returned information into natural language. A result might include a bear’s name, distance, mood, or message.

Tool calls are useful wherever a system needs current information, computation, or an external action. Web or database search can fetch information not available in the model’s training data. Weather and stock tools can answer questions about changing values. A calculator can perform arithmetic deterministically; the lecture contrasts this with relying on next-token prediction to multiply two numbers. Code execution can produce a plot or run a computation. Other tools can send messages or carry out computer actions.

The tool interface also affects latency. A location lookup may be nearly immediate, while a web or backend search may be noticeable. The lecturer’s view is that such calls are often not the dominant part of total latency compared with generating the answer, though the timing varies with the action.

The model can learn tool use through supervised examples that show the tool call, its arguments, and the response to generate from the backend result. The lecture describes this as an earlier approach to teaching the model a fixed sequence of tool prediction and response generation. More commonly, it says, tool instructions are supplied through prompting rather than being baked into the model’s weights. Well-designed APIs can reduce how much explanation is needed, but the model still has to be told what tools are available and how to use them.

Writing those descriptions is itself a design problem. A proposed route is to use a reasoning model to revise tool documentation against example queries and expected answers. The system compares the current model’s actual responses with target responses and iterates on the tool descriptions to move the outputs closer to the expected ones. The method depends on having useful examples and labels; the lecture warns that these can be difficult to obtain, and that synthetic answers may not match what the application actually needs.

A system with many functions creates a selection problem. A request such as “Show me my teddy bear’s morning routine” might be answered by a tool that retrieves a document from a drive or by one that searches another document store. If both return the same result, the overlap is benign. But “Can you find me my teddy bear?” could mean locating a photo on the user’s computer or finding the physical bear nearby. Here, tools with overlapping scope can lead to different interpretations of the user’s intent.

The lecture takes seriously the objection that such a request may simply be underspecified. Its response is that users should not bear the whole cost of ambiguity when the system exposes tools they may not know are active. If outputs repeatedly diverge from the user’s intent, that can indicate the tool set has grown too broad. Setting a model’s temperature to zero might improve repeatability, but the lecturer cautions that lower randomness can reduce answer quality—and can make the same wrong interpretation happen consistently. Determinism alone does not resolve a mismatch between a request and the tools available.

Tool selection narrows the available set before the main model makes a tool call. A router can use the conversation history, long-term memory, user profile, and settings to choose which function APIs to place in context. If the conversation has been about a physical teddy bear, that history may favor a location search. Past preferences may help resolve an underspecified request. In an enterprise context, a user’s role might help choose between finance and engineering tools. User settings matter too: a system can respect an inactive tool or ask whether the user wants to enable it.

The router need not be the most expensive model. The task remains in natural language, but the lecturer describes the routing model as typically lightweight and low-latency. Routing policies can also exclude tools that overlap: whether two tools are selected together depends on the use case and the rules attached to the router.

There is a separate cost to exposing too many tools: context bloat. Even if the tools do not conflict, their definitions take up tokens, and the lecture connects larger contexts with degraded performance in retrieval and tool calling. It is therefore useful to minimize what enters the prompt, not only to control token costs but to avoid making the model search through an unnecessarily large set of instructions.

Model Context Protocol (MCP) addresses a different problem: duplicated integrations. Without a common standard, a tool may need separate implementations for different LLM providers. MCP defines a way for clients and servers to communicate across supported systems. The server contains tool implementations and business logic; the client connects the user-facing application to that server.

The lecture groups MCP capabilities into tools, prompts, and resources. Tools define operations and their inputs and outputs. Prompts provide handles or shortcuts for workflows that use tools. Resources make information available to the application layer. Its book-recommendation example separates a function for finding a particular poetry title from one for recommending books, gives each a workflow prompt, and makes the teddy bear’s existing collection available as a resource. Standardization can make these pieces reusable across clients, but it does not automatically solve the question of how to write effective tool descriptions or decide which tools a model should use.

The tools and instructions also have a token cost. Prompt caching can reduce the cost of repeated fixed content. The lecture’s practical rule is to put stable material at the beginning of the prompt, before changing user queries or other variable context. If the tool definitions remain the same across requests, frontloading them allows the repeated prefix to be reused from cache. A change to a tool description requires the cache to be refreshed, but descriptions may change less often than requests arrive. The lecture describes cached inputs as substantially cheaper in the pricing examples shown, with the exact savings and cache duration depending on the provider and configuration.

A model is only one part of an agent. The lecture uses harness to mean the surrounding system: tools, the order in which actions happen, prompting, guardrails, memory, and strategies for managing context. Its analogy is a car: the model is the engine, the harness is the rest of the vehicle, and both are needed for a usable agent. A strong harness cannot make a weak model sufficient for every task, nor can a capable model operate effectively without a system around it.

ReAct—Reason plus Act—provides a way to describe an agent’s loop. Given an input, the agent observes what is known, plans what to do, acts through a tool or other mechanism, and observes the result. It can repeat that cycle until it has enough information to respond. The lecture’s thermostat example makes the loop concrete: a request says the teddy bear is cold; the agent identifies that the room temperature is unknown, plans to check it, calls a temperature function, and observes a reading of 65°F. It then plans to raise the temperature by 5°F, calls an adjustment function, and observes that the thermostat is set to 70°F before replying.

The important feature is not a particular number of steps. The agent must incorporate what its actions returned before choosing what to do next. The lecturer distinguishes this needed recap and reasoning from the additional reasoning associated with reasoning models: the observe–plan–act pattern can be part of a non-reasoning model’s tool-use behavior as well. The stages are largely learned through training, but they also describe the operational structure that the harness must support.

When a task outlasts the available context window, the system must compact the conversation. Deletion can remove tool outputs that were only needed for the next turn. A sliding window can keep recent exchanges on the assumption that recent material is more relevant. A third approach summarizes what has happened into structured fields and continues with that shorter account. The lecture gives examples such as current progress, constraints, remaining work, and references. These approaches can be combined.

Compaction is about preserving enough of a conversation to continue working; memory is about keeping information available beyond that conversation. “Short-term memory” is simply the current context window. Long-term memory can preserve a useful preference or fact for a later interaction—for example, that the user thought a teddy bear might make a good gift. One method is to retrieve relevant passages from prior conversations using RAG. Another is to use a memory tool that stores information when the agent detects a cue to save it, with the saved material made available in future context. The lecture suggests that compacted state is usually a high-level account of what the user is trying to do, not a verbatim preservation of reasoning traces.

Coding agents apply the same action loop to an environment that can read and change files, run shell commands, use the web, and call other tools. The lecture’s architecture diagram places an agent loop between the user’s task and the tools, with an execution environment, a permission system, and state persistence around it. The loop proposes actions, receives results, and continues from the state those results produce.

The permission system governs whether proposed actions are allowed, denied, or sent to the user for approval. That distinction matters because an agent’s actions can change files or run commands in the user’s environment. Earlier systems often asked users to allow or reject every action. The lecture says that this has become less common: a lighter model may now evaluate proposed actions and automatically approve some of them. The permission layer remains a distinct part of the system, even when it is less visible to the user. In other words, the agent loop does not simply execute every action it proposes; the harness can place a safety decision between the proposal and the tool call.

Coding agents often retrieve information differently from a conventional RAG pipeline. Rather than searching primarily for semantic similarity, they can use grep and related patterns to find exact terms and code locations. A model can generate variations of a symbol or request and search for them; the results then feed its next reasoning step. The lecture describes this as efficient and says grep appears frequently in coding-agent workflows.

Project-specific rules can persist in files such as AGENTS.md. Such a file can tell the agent how to navigate a repository, install a package, run tests, or check the continuous-integration plan. It gives users a way to state preferences and constraints that should apply across sessions, rather than having to repeat them in each prompt.

The diagram’s state-and-persistence component is where the system can preserve information needed to continue work. That is distinct from the immediate results passed around inside a single agent loop. A coding task may involve multiple actions—inspect files, edit code, run tests, and respond to failures—so the agent needs some account of what has already happened and what remains. Project instructions can shape how it behaves across sessions; persisted task state helps the system resume or track work. The lecture presents state persistence as part of the coding-agent architecture, alongside the loop, tools, execution environment, and permissions.

Reusable procedures can be captured as skills. A skill is a recipe for a recurring workflow, potentially combining instructions, scripts, documentation, and other assets. Instead of writing every skill from scratch, a user can ask a coding agent to turn an end-to-end workflow into a reusable procedure. Skills make the prompting and execution process repeatable; they do not remove the need for tools or for the agent loop that uses them.

Computer use is another developing capability: an agent can take screenshots, simulate keyboard and mouse input, edit files, and browse the web. The lecturer describes this mode as inefficient because the system has to interpret screenshots and predict subsequent actions, and points to later course material for possible improvements. It is a distinct route to interacting with software, not a substitute for more direct tools such as shell commands or file editors when those are available.

Agents can also delegate. In a subagent pattern, a main agent assigns specific tasks to separate agents, which work and return results without communicating with one another. This can make independent work easier to divide, but each subagent needs its own context and agent loop, which adds cost. In an agent-team pattern, agents can share a task list and communicate while working. The lecture notes both the promise of collaboration and the risks that come with giving agents substantial power, pointing to examples of reported progress as well as security concerns. Delegation expands what a system can do, but it also expands the work of managing context, permissions, and coordination.