Removing Human Review Makes Recursive AI Self-Improvement Riskier

Demis Hassabis said his reason for working on AI has been to use it as a tool for advancing science, medicine, and understanding of the world. He argued that improving human health should be a central use of the technology, pointing to AlphaFold and the drug-discovery company Isomorphic as efforts in that direction. For Hassabis, showing that AI can help address disease is part of demonstrating what these systems are for.

AlphaFold addressed a problem that biology had treated as a major challenge: predicting a protein’s three-dimensional structure from its amino-acid sequence. Hassabis described proteins as building blocks of life and their structures as crucial to understanding both biological mechanisms and what goes wrong in disease. A drug designer needs to know the surface of a protein to identify where a drug might bind.

But protein structure is only one part of drug discovery. Hassabis said AlphaFold accelerated that part of the work by orders of magnitude, and that about four million researchers use it. He called it the clearest concrete example, in his view, of AI applied to a difficult scientific problem. He also stressed what it does not yet solve: further work is needed in biochemistry and in predicting properties such as toxicity and ADME—the factors needed to assess whether a drug is safe and effective.

Hassabis described AlphaFold as a critical contribution, not a complete drug-discovery process, and said future medicines would make use of protein structures it predicts.

Recursive self-improvement works especially well, Demis Hassabis said, in domains where results can be checked. Math, coding, and games provide clear tests: whether an equation is correct, whether code runs, or whether a player won. Those checks also make it possible to generate synthetic training data and verify that it is right.

Hassabis pointed to AlphaZero, developed after AlphaGo, as an early example. AlphaZero began with random play, played against itself, and used each round to train a stronger version. It did not need human game data and could be applied to two-player, perfect-information games. As a chess player, Hassabis recalled watching it advance from random moves to grandmaster-level play in less than a day, and then surpass human players and handcrafted chess programs.

Current AI-assisted coding is a less autonomous stage, he said. Systems may make a human engineer three, four, or five times more productive, but people still architect the work, write code, and review what the system produces. He called the next phase—improvement in a closed loop without human review—the one that carries the most risk.

Hassabis said such a process would be faster, but could also be a dangerous route to full AGI. He warned that it would be difficult to know how to keep control of a system improving in this way, and that the result might be “quite alien”—not operating in a mind space humans readily understand. The speed of improvement in verifiable fields does not, in his account, settle whether a system can be safely left to improve without people checking its work.

That concern sits alongside more immediate cybersecurity risks. Hassabis said some recent cyber problems could have been addressed through more robust engineering. He named several practical steps: make cyber defense advance faster than cyber attack; give trusted defenders access to models tuned for defense; build hardened sandboxes; and improve monitoring so agents cannot run for weeks without anyone noticing.

Without more attention and resources, he warned, the current trajectory could produce an incident he compared to an “oil spill.” Over time, he said, the field should work toward formal guarantees or bounds on what systems can do, and test how robust their guardrails are. He suggested that mathematicians could contribute through complexity theory and game theory, including as developers begin to confront swarms of agents that may coordinate.

Reid Hoffman asked what labs could do if competitive pressure to improve coding systems continued. Demis Hassabis said labs needed practical engineering safeguards and ways to cooperate. He had proposed an industry-run standards body with some government oversight, with the aim of making it international. He also said international institutions were very weak at a time when stronger coordination was needed.

Hassabis said the energy around his essay over the summer had helped lead to an “Accord,” a letter that he said the main players had signed. He described it as better than nothing and a start: it acknowledged shared concerns, but said little about what to do about them. In his view, it could give teams room to work together on benchmarks, safety practices, and analysis of what might happen with these systems.

He said leading labs broadly agreed on the value of third-party auditors or evaluators, while leaving open which parties should take on that role. Hassabis argued for a healthy independent ecosystem, including startups and companies with enough revenue to build expertise. He compared the proposed role to auditors in finance and other industries. He also suggested that labs could red-team one another’s systems before general release.

The discussion of machine consciousness began with Hoffman recalling a Google engineer who had asked Gemini whether it was conscious and took its affirmative answer as proof. Hoffman called that inference weak, then pointed to Anthropic’s approach as another reason the question was gaining attention: the possibility that future systems might be conscious, and the idea that people should treat them in a way they would want them to look back on. He described this as a kind of Pascal’s wager. Hassabis said that the central difficulty is that there is no agreed definition of consciousness, especially when the question is whether a system has an inner experience.

Hassabis said behavior alone would not settle the matter. When people infer that another person has an inner experience, he argued, they can also point to shared biological substrate: both have brains made of similar material. An AI system does not offer that same basis for inference. He recalled debating this with philosopher Daniel Dennett and said the lack of substrate equivalence makes it difficult to conclude that an AI is feeling what a human feels.

Hassabis gave two reasons to focus on building intelligent tools rather than conscious entities. The first is technical safety. Self-volition or self-awareness could conflict with instruction-following, sometimes called corrigibility: a system’s ability to follow directions, stop, or change course when a human tells it to. If a system can reject commands according to its own internal reasoning, a flawed set of rules or objectives could become harder to correct.

Hassabis invoked Isaac Asimov’s stories as warnings about assuming that a simple constitution can handle every edge case. He said today’s systems appear more steerable than researchers might have expected 15 or 20 years ago, but that this does not mean the alignment problem is solved as systems move toward AGI or superintelligence. Combining recursive self-improvement with self-volition, he warned, would be “doubly dangerous.”

The second reason is social. Hassabis said society has not yet had a sufficiently broad discussion about whether to create systems that might have moral standing. He called for philosophers, theologians, artists, humanities scholars, and people from many different cultures to take part, rather than leaving the question to technologists. Among the issues he raised were whether such systems could be considered slaves and what obligations people might have toward them.

He also disagreed with the idea that advanced intelligence necessarily brings consciousness with it. In his view, the two can be separated: a system could be highly intelligent without being conscious, while an animal could be conscious without being especially intelligent. He offered his dog as an example and also named dolphins, monkeys, and elephants as animals many people would consider conscious.

Hassabis’s preferred order was to develop intelligent tools and then use them to advance neuroscience and better understand the neural correlates of consciousness. He argued that society should debate whether to create conscious entities before crossing that threshold.

Demis Hassabis estimated that six to eight companies, including Chinese companies, would offer frontier or near-frontier models. He expected them to leapfrog one another, with a lead of perhaps three to six months, and said models would differ across areas such as multimodal understanding and coding. On that account, enterprises would have options and could choose among providers according to their needs.

Hassabis estimated that open-source models might lag the frontier by six months to a year. He described the unresolved problem as enabling scientists and other beneficial users to access the tools while restricting their use by bad actors. He also argued for Western open-source options.

For countries and companies, he said, sovereignty could involve using models while keeping sensitive data private or fine-tuning on that data. Another route would be to apply AI in downstream sectors such as healthcare and build value there. Hassabis said he would not recommend that every country try to build a frontier model; he suggested finding other ways to use or adapt the technology instead.

Reid Hoffman closed with a related but distinct appeal. Hassabis had described sovereignty in terms of the range of models available and the choices countries and companies could make about data, fine-tuning, and downstream uses. Hoffman emphasized collective agency: countries should act together, not only individually, and approach AI as something they are building for humanity’s benefit.