Orply.

Language Models Begin With Language, While Human Agency Begins With Cooperation

Tim ScarfeMichael TomaselloMachine Learning Street TalkThursday, November 19, 202617 min read

In *The Evolution of Agency*, developmental psychologist Michael Tomasello argues that agency is an organization for making and revising decisions when the environment cannot be predicted—not simply a measure of intelligence or learning. He traces its development from basic goal-directed control to human capacities for planning and shared goals. Those social capacities, he argues, underpin language, and large language models have begun with language’s endpoint while missing much of the coordination that made it possible.

Agency is a response to what cannot be predicted

Michael Tomasello starts with environmental uncertainty. When conditions are predictable, he argues, evolution can build essential behavior directly into an organism: breathing and swallowing do not require a decision maker to assess each situation afresh. Decision-making becomes useful when no fixed response can cover the possibilities.

That is especially true when uncertainty comes from other agents. Prey, predators, competitors, and cooperators cannot fully predict one another. An organism needs a way to assess what is happening and choose behavior that fits the situation. Nature cannot anticipate every contingency in advance, Tomasello says, so it builds an organization that lets the organism respond “on the ground.”

His concrete model is a feedback-control system: an organism has a goal, perceives whether the world matches it, acts to change the world, and registers the effects. If an action does not bring about the goal, the system can try something else. In the HVAC diagram he shows, a thermometer measures room temperature, a comparator checks it against a target of 72 degrees, and heating or cooling changes the room. The example makes the organization visible; Tomasello is not claiming that a thermostat is an agent in the full psychological sense.

A traffic light illustrates the difference between a fixed response and flexible control. A timer changes the lights whether or not cars are waiting. To make the system responsive, Tomasello says, it needs to perceive the cars, have a goal for the intersection, and choose what to do in light of both. He offers balancing the number of cars on the intersecting roads as one possible goal. The point is that perception, a goal, and a mechanism for choosing an action have to work together.

A robot vacuum adds layers of goals and subgoals. In the schematic Tomasello presents, the machine’s goal is to fill its hopper. It searches for a leaf, moves toward it, vacuums it up, and checks whether the hopper is full. Each activity can involve smaller feedback systems, such as moving and aligning the wheels. When Tim Scarfe says something still seems missing before he would call the vacuum an agent, Tomasello accepts the distinction. He is using these machines to identify a basic organization of perception, action, cognition, and goals—not to claim that they possess a mysterious inner decision maker.

That distinction also explains why Tomasello resists defining agency by complexity. An ant colony can produce sophisticated behavior, and a spiderweb can be complex, without demonstrating that an individual organism controls flexible action in response to its immediate situation. At an AI symposium, he recalls, some participants treated being smarter as necessarily advantageous. His counterpoint is that comparatively simple organisms can do well because they are adapted to their environments. What matters for agency is control over flexible action, not complexity on its own.

Nor does agency require freedom from biological causes. Tomasello is not arguing for free will untethered from biology. An organism’s adaptations constrain it, but within those constraints it can choose among possible actions. That distinction informs his answer to whether machines could become more agentive than people. He does not rule it out, especially as robots, drones, and autonomous vehicles increasingly perceive and act in the world. Their organization can be structurally similar to an agent’s, he says, but he is unsure what is missing.

One candidate is an executive tier of organization: a system that monitors its own attention and action, anticipates mistakes, plans, and tests hypotheses. A further, metacognitive tier would let it assess its plans, consider alternatives, or recognize that it needs more information. Tomasello treats these not as isolated skills added to an otherwise unchanged system, but as additional levels of organization.

He would like machines to be able to form novel goals and strategies when familiar ones do not fit. Goals supplied by humans do not, by themselves, disqualify a machine from being agentive in his account: humans, too, begin with biological goals. The harder question is whether a system can formulate a new goal when it encounters a situation for which its existing goals and strategies are insufficient. Tomasello is not alarmed by the prospect of AI taking over the world; he has heard such predictions since the 1970s and describes AI as an important tool. His proposal is to study natural agents as models for building more agentive systems.

The proposed evolutionary tiers add control, not just complexity

Tomasello presents the evolutionary sequence as increasingly elaborate ways of directing and controlling action. He treats present-day reptiles as models for early land vertebrates, while cautioning that a model species is not identical to the ancestor it stands in for. The levels are his proposed account of the line leading to humans, with birds set aside as a separate evolutionary track.

LevelOrganization Tomasello associates with itIllustrative capacity
Goal-directed agencyFeedback control; goals direct action and attentionActing to bring the situation into line with a goal
Intentional agencyAn executive tier, including working memory and plansMonitoring uncertainty, inhibiting an action, and testing a possible strategy
Rational agencyA second-order executive tier, or metacognitionAssessing a plan, seeking more information, or choosing a safer option
Normative agencyShared goals and plans, with social and normative self-regulationCoordinating roles and regulating action in relation to shared expectations
Tomasello’s proposed organization of agency, from feedback control to human shared agency

In Tomasello’s account, reptiles make simple decisions through basic control systems. Mammals add an executive layer. A squirrel deciding whether to jump to a branch, for example, may appear ready to jump and then take another route, as if it can represent the possibility of falling. Planning requires representing states of the world that do not yet exist, generating hypotheses, and testing possible actions before committing to them. Tomasello’s formulation is that the animal’s hypothesis can die instead of the animal: if the plan looks likely to fail, the animal can abandon it first.

He calls this intentional agency. Intentions and plans let an animal abort an action before carrying it out. Tomasello also points to squirrels devising ways around backyard bird-feeder obstacles as examples of flexible responses to novel circumstances. The examples illustrate strategy generation and testing, not a claim that squirrels outperform people on a general measure of intelligence.

The next level, rational or metacognitive agency, is clearest to him in great apes. An animal may delay a decision, seek more information, or choose a safer option when it recognizes that a problem is difficult and error is likely. It can monitor its decision process and change its approach, rather than merely trying a different action.

Tomasello proposes that different social and ecological pressures help explain these levels. He describes reptiles as mainly solitary, with much of their uncertainty arising from prey. Group-living mammals also compete with groupmates for food and may have to act faster than their competitors. Great apes face more contest competition, in which they must outsmart one another rather than simply arrive first. These are parts of his evolutionary account, not a claim that each species fits neatly into a single category.

Learning matters, but Tomasello argues it is not the whole story. Learning helps an agent make adaptive decisions; it is a subprocess, not the definition of agency. Some organisms can behave appropriately in a situation without learning in that situation, while most organisms also retain experience for future use. His criticism of approaches that emphasize learning, including reinforcement learning, is that they can leave the agent’s organization in the background. He draws an analogy with biology: living things are made of the same basic elements as non-living things, but organized differently. For agency, too, the question is what kind of organization connects goals, perception, and action.

That emphasis helps explain why Tomasello is wary of “intelligence” as a general measure. He traces the concept to Alfred Binet’s work in a French educational context, where the task was to determine whether children could benefit from ordinary instruction. Tomasello worries that the term remains too closely tied to school-oriented measurement and verbal or abstract performance. Squirrels can remember more hiding places than humans; some birds discriminate more sounds; chimpanzees can outperform children on some tool tasks. Different animals excel at different things.

Tomasello does not deny specialized mechanisms, such as those involved in foraging or mate choice. His disagreement is with the idea that everything is modular. Perception, attention, learning, and memory cut across domains, and an agent has to resolve conflicts between competing motives. A foraging mechanism and a mate-choice mechanism might each be specialized, but neither alone settles what the organism does when their demands conflict.

Emotions influence those decisions rather than replacing the decision maker. Fear of a snake can strongly push an organism toward leaving; attraction, anger, or happiness can similarly weigh on what it does. Tomasello calls this nature “putting its finger on the scale”: adaptations make some responses more compelling but do not determine every action. His deliberately stark example is that someone with a gun to their head might let a snake crawl over their foot.

Social commitments can pull in different directions as well. A person might have promised to meet someone but break that promise to help a daughter who has been in a car accident. Someone may be drawn to sugary food but decide not to eat it while on a diet. Tomasello’s argument is not that biology or society leave the individual untouched. Both influence the decision; the agent has to resolve competing pressures in the situation.

He uses the same reasoning against accounts in which genes, social class, or ideology act as the only real causes. Natural selection and cultural conditions shape what people are disposed to do, but, in his view, they do not settle every choice. He prefers the language of cognitive capacities, knowledge, planning, foresight, and hypothesis testing to “general intelligence.” The control-system model points to an organization that can be specified: how an organism controls action, what information it uses, and what lets it reconsider a decision.

Human agency is built between people

The distinctively human step in Michael Tomasello’s account is not simply a higher level of individual skill. It is shared agency: people commit to a joint goal, take on roles, and understand that their actions have to fit together. If one person fails to perform a role, the other can object because the failure threatens something they are pursuing together. The commitment makes each participant answerable for the joint activity.

Humans not only use language, they created languages. They created them.

Michael Tomasello · Source

Joint attention is part of what makes shared agency possible. Cooperators must attend not only to an obstacle or opportunity, but also to the fact that they are attending to it together. They need to know that the other person has noticed it, so they can form a unified strategy. For Tomasello, this capacity supports a distinctively human grasp of perspective: two people can attend to the same object while understanding it in different ways. A dog can also be an animal, a pet, or “it”; which description matters depends partly on what the speaker and listener are trying to communicate.

Tomasello presents this capacity as socially developed. A child raised in isolation, he suggests, would not acquire the same notion of perspective, because there would be no other person’s contrasting view to take into account. The possibility of treating one view as one among others grows, in his account, out of social engagement.

In experiments comparing children and great apes, Tomasello says, differences show up in how children maintain collaboration. Children can continue working together after one has received a reward, while chimpanzees tend to stop once they have obtained theirs. Children also correct one another when a partner does not play a collaborative role properly. For Tomasello, that correction shows more than two individuals acting in parallel: each partner is regulating behavior in relation to a shared goal and shared expectations.

Shared agency scales up into cultural life. People inherit conventions, norms, and institutions they did not invent individually. These structures constrain conduct, but they also provide tools for doing things that would otherwise be difficult: a shared language, for example, can be used to write poetry. Tomasello treats culture not as another rung in a simple ladder of individual agency, but as a different form of agency built from shared commitments and common practices.

His experiment with children communicating by video illustrates how conventions can form. When researchers cut the sound and asked children to gesture to one another, the children invented gestures and began using the same ones. They adapted to each other and conventionalized the gestures. They did not start with a shared set of words for the task; they established a way to communicate through interaction.

That origin story informs Tomasello’s challenge to language-first approaches to AI. Large language models are useful tools, he says, but to his knowledge they do not create linguistic conventions or engage in discourse around topics in the human sense. They begin at what he calls the endpoint: language, a pinnacle of human achievement, without the shared-agency infrastructure that made language possible. He allows that it may be possible to skip that infrastructure. But he thinks building from coordination, shared attention, and perspective-taking is a more promising route to machines with human-like agency.

The coordination problem comes before the words. Tomasello uses the stag hunt: two people can pursue a valuable animal only if both cooperate, but either may be left with nothing if the other does not join. Each wants to proceed only if the other does. The situation resembles two people caught in an “after you” exchange: neither can get the desired result alone, and each must take account of what the other will do. This requires more than individual cleverness; each has to consider the other’s perspective, including what the other thinks about their intentions.

In this account, conventions are coordination devices. A shared convention helps people align their actions without inventing a new solution every time. Tomasello therefore recommends working on the bottom-up problem alongside work on mature language systems: how agents coordinate, take one another’s perspectives, and create conventions. He does not claim that language models can never develop such capacities. His proposal is to investigate coordination and convention-building rather than assume mature language supplies them.

Perspective-taking also helps explain features of language that can seem merely grammatical. “It was Fred that broke the window” can correct a listener who thinks someone else did it. The construction is useful because the speaker is adapting the message to a mistaken impression. Active and passive sentences can present the same event from different perspectives: “The dog is chasing the cat” and “The cat is being chased by the dog.” The event has not changed, but the sentence changes which participant is foregrounded.

Grammar, in this view, is not only a system for combining words. Its constructions are conventions for presenting things in ways suited to a listener and a context. Calling something “a dog,” “an animal,” “a pet,” or “she” depends on what is relevant in the exchange. If the point is that animals are not allowed in a building, “animal” may do the work; if the speaker is advising someone who wants a companion, “dog” or “pet” may be more useful.

Tomasello and Scarfe differ in emphasis over what language models can learn from text. Tomasello suggests that discourse carries logical connections and shifts in perspective that contribute to a model’s capabilities. He asks what would happen if every sentence on the internet were randomized, stripping away coherent texts and their causal or logical progression. At a workshop, he reports, the consensus was that models trained on that material would perform much worse. He presents this as a question about the contribution of discourse, not as an established experimental result.

Scarfe raises a different possibility: a model might learn from randomized language, even if the result were meaningless to people. His concern is that language models do not produce the kind of agent-centered storytelling organized around a character’s experiences, goals, and conflicts. Tomasello agrees that this points back to the infrastructure he has described. The distinction is not simply whether a model can reproduce grammatical forms, but whether its use of language is grounded in taking account of agents, their perspectives, and what they are trying to do with one another.

Pointing offers another example of communication grounded in shared context. A gesture can mean “there is the thing we need,” “your sister is over there,” or “the library is open,” depending on what the participants know. The listener has to infer why the speaker is directing attention to that object. Tomasello glosses the gesture as “look over there and you’ll know what I mean.” He acknowledges that it may not be fair to say an LLM cannot interpret pointing, and that the infrastructure might be skipped if the aim is simply to build a useful tool. For a system with agency, he thinks it is worth investigating how coordination, perspective-taking, and simpler forms of communication can support convention-building.

Shared realities can coordinate people—and be exploited

Shared meaning lets people build entities that exist because people collectively recognize them. Michael Tomasello uses money and political office as examples: a piece of paper can function as money, and a person can hold the office of president, because people participate in systems that assign those roles. Such facts depend on human concepts, but they are still facts about the world. Corporations, too, are not individuals in the ordinary physical sense, yet they can be legally recognized entities with substantial power.

Tomasello describes these as cultural creations, drawing on the idea of status functions: people make a person into a president or a piece of paper into money. The status depends on shared recognition and institutions, but it can have material consequences. A currency that once functioned as money can become merely paper when the system that sustains its status changes. An office can be a human-relative fact—one that exists because people have a concept of it—without being unreal.

Those shared realities can be distorted. Tomasello describes how repeated claims and appeals to what “everybody” thinks can exploit people’s tendency to use others’ apparent beliefs as a guide. If someone repeatedly says they are hearing that a view is widely held, listeners may take that as evidence that others in their community believe it. Repetition can make the claim feel familiar and socially established. Tomasello connects this to the possibility of creating an alternative reality by exploiting the same social capacities that make shared understandings possible.

Lying also relies on cooperation. Tomasello argues that a listener ordinarily assumes a statement such as “it’s raining” is meant to convey information, not deceive them. He says chimpanzees do not lie to one another in the human way, because the expectation of a truthful, cooperative response is what makes deception work for people. His point is that a general expectation of cooperation can make lying effective.

Cooperation also has a group boundary. Humans can form a strong sense of “we” with people who look, speak, eat, and dress like them, while regarding another group as unfamiliar or suspect. Tomasello calls this a dark side of collective agency: people may cooperate closely within a group while acting against another. He does not describe the boundary only as hatred; it can also be uncertainty about people who seem different. Modern societies require groups to manage their interdependence, he says, and shared intentionality may help people adapt—but the challenge is urgent.

This account does not mean that everything produced by social life is itself an agent. Scarfe asks whether a market might be better understood as a superorganism than as a collection of separate decision makers. Tomasello is wary of treating capital as an agent. Capital may appear to flow, but he argues that capitalists make decisions about where to direct it. He is unsure that capital itself has goals or adjusts its conduct when those goals are unmet.

Tomasello does accept that social patterns can emerge from individual actions without being intended by any one person. A traffic jam is his example: everyone is trying to get home, and their separate intentions produce an outcome no one sought. Inflation and market crashes, he says, can similarly emerge from intentional decisions without being anyone’s intention. The “invisible hand” is a way of talking about such outcomes, but he distinguishes an emergent pattern from an agent that intends it.

Scarfe presses the alternative: perhaps a market is best analyzed as an agent rather than as a collection of individually rational agents. Tomasello accepts that agent-like language can be useful. His caution is about treating the description literally. Does capital have a goal? Does it adjust when it fails to reach it? He is unsure. Explaining individual decisions is not the same task as describing a sociological pattern that emerges from them.

That distinction matters for policy. If harmful outcomes arise from the interaction of individual choices, policymakers may need to recognize the pattern and regulate the choices that generate it. Tomasello names collective-action problems, inflation, traffic jams, and stock-market crashes as outcomes that policymakers should connect back to individual decision-making. He does not deny the usefulness of describing markets or capital in agent-like terms; he cautions against treating that description as proof that the emergent pattern itself has intentions.

Conversation alone would not establish agency

Whether a system can behave as if it has agency without possessing it takes Tomasello into territory he calls metaphysical. He invokes Daniel Dennett’s “intentional stance”: explaining or predicting a system by attributing beliefs, desires, and intentions to it. Tomasello does not claim to resolve whether such descriptions reveal real agency. But if a machine displayed the relevant qualities—goal-directed control, executive monitoring, and the other capacities he has outlined—he thinks he would have to regard it as some kind of agent.

A Turing test, he says, would not come close to being sufficient. Human-like conversation alone would not establish the deeper organization he has in mind: whether a system has goals it pursues, monitors its own actions, or can revise a plan when its expectations fail.

At the same time, Tomasello has no settled answer to whether physical substance matters: whether agency or consciousness requires flesh, hormones, and neurons of the kind humans have. He recalls a discussion of whether “meat” matters, without a consensus among those present, and offers no definitive view of his own. If a machine had the organization and capacities he describes, he thinks he would have to treat it as some kind of agent, while leaving open whether it would be a human agent or whether its biological differences would matter.

The frontier, in your inbox tomorrow at 08:00.

Sign up free. Pick the industry Briefs you want. Tomorrow morning, they land. No credit card.

Sign up free