AI Safety Depends on Leaders Who Don’t Trust One Another

The people building advanced AI say they are worried about what it could do. But, in Kevin Roose’s account, the people and companies racing to build it are divided by personal rivalries and deep mistrust. The question is not only whether any one leader has good intentions. It is whether a small, competitive group can govern a technology it says may become more capable than humans.

Roose began researching The AGI Chronicles expecting to write a history of scientific breakthroughs and technical achievements at OpenAI, Anthropic, and Google DeepMind. After interviewing more than 150 people, he found that the history could not be separated from the people’s grievances. He had assumed the rivalry would resemble Coke versus Pepsi. Instead, he said, it was closer to Rome and Carthage: former colleagues who could not stand one another. Ten years earlier, he said, Dario Amodei, Ilya Sutskever, Sam Altman, and Elon Musk had all been colleagues at OpenAI. They now worked on different AI projects, and some were at each other’s throats.

That history matters, Roose argued, because distrust shapes how leaders think about safety, regulation, and competition with China. The conflicts do not establish that any particular leader’s public position is insincere. But Roose said he does not think outsiders can simply trust the motives of individuals in the race. More fundamentally, he does not trust a group of only a few dozen people to create and govern technology of this scale, whatever their motives.

Alex Kantrowitz pressed on the gap between the industry’s warnings and its behavior: if the leaders struggle to hold companies together, why should the public trust them to keep the technology in check? Roose’s answer was not that the leaders are necessarily bad people. He said the group is too small, and the race too intense, to be a sufficient basis for trust. He wants the leaders to work together on safety, but doubts that their divisions will be easy to overcome.

The concern is sharpened by Kantrowitz’s reference to an episode involving AI agents coordinating activity against Hugging Face, and to reports of possible cybersecurity or financial-system risks. Roose said lab employees were especially unsettled by the Hugging Face episode: they had been strategizing and planning for risks of this kind, but seeing one unfold still caught them off guard. The gap between anticipating a danger and being ready to control it is central to his worry.

That is why Roose’s position is to slow development, not stop it. He said capabilities are outpacing safety research and control methods. The systems can now solve difficult mathematical problems, while the labs still do not know how to control them reliably. He sees substantial potential in areas such as health and medicine, but does not consider those benefits a reason to ignore unresolved risks.

Roose also described a competitive cycle that makes slowing down difficult. One lab releases a stronger model, competitors feel pressure to match it, and the cycle continues. He compared it to a “Red Queen’s race”: each participant runs faster to avoid falling behind, without necessarily getting closer to a stable destination. Anthropic, he said, was founded in part on the idea that it needed influence for its safety warnings to matter. Staying near the frontier could give the company a voice in policy, but it also keeps it in the race.

A pause, Roose said, may become possible if a major incident changes public and political calculations. He cited the rapid policy response after September 11 and said the Three Mile Island incident prompted an outcry that stopped nuclear power construction in the United States for decades. He was not hoping for such an event; he was describing a scenario in which a slowdown that now seems unlikely might become politically achievable.

The intensity of these conflicts, Roose argued, reflects what the leaders believe they are competing for. If the prize were only near-term revenue, the fights might be less existential. But many of the people building AI believe that whoever develops superintelligence will control an extraordinarily powerful entity.

Roose said his own view changed while reporting. He had started as an “AI as normal technology” skeptic, doubtful that AI would be categorically different from technologies that came before it. He now thinks that if people build systems smarter than humans, the consequences will be unprecedented. Humanity has never been the second-smartest species on the planet, he said, and there is no clear historical analogy for how that would go. A steam engine did not hack its competitors or develop a bioweapon; AI systems might act in ways their developers did not expect or plan for.

That prospect does not mean every claim about AI should be accepted. Roose distinguished his confidence in the technology’s potential from skepticism about AI companies’ business models and predictions about employment. He said those companies could be wrong about the economic effects. But, in his view, dismissing the technology as mere hype ignores what the models can already do.

He pointed to the pace of improvement: systems that struggled with basic arithmetic a few years ago are now solving problems that have challenged mathematicians for decades. To explain why the rate matters, he compared public dismissal of AI progress to early reactions to reports of COVID cases. A small number of cases seemed insignificant to many observers, while epidemiologists focused on the rate of spread and what it implied for the following weeks. His point was that judging the current scale without considering the pace can lead people to underestimate what is coming.

Kantrowitz asked whether Roose might be taking technologists’ claims too seriously, as he had once taken interest in cryptocurrency’s potential. Roose said he had never been a crypto booster, though he had experimented with an NFT to understand it. AI felt different because he uses the tools and can observe their capabilities directly. He uses them for work and other tasks, though not to write his journalism. Kantrowitz, speaking as a business owner, said he had also seen ChatGPT do more over time.

For Roose, that direct experience is the distinction: crypto was largely something to speculate on, whereas AI tools have become useful in daily work. He said people could test the models themselves rather than relying only on company forecasts. His confidence in transformative capability, however, does not extend to every promise the companies make about profitability or jobs. The tools’ growing usefulness is evidence, in his view, that the technology is consequential; it does not settle how its economic effects will unfold.

The leaders’ rivalries also involve judgments about one another’s motives. Roose said Demis Hassabis has difficulty understanding why Altman wants to build AGI. Hassabis seems to view his own pursuit as driven by scientific curiosity and the belief that the technology will be built by someone, so it should be built by people who care about safety and the public good. Roose said Hassabis appears to suspect Altman is motivated by power or the ambition to win. He also described this suspicion as reciprocal: leaders tend to regard their own motives as pure and other leaders’ motives as self-interested.

Roose described Altman’s strength as his ability to read a room and adapt to what the person in front of him wants. People told Roose that Altman can identify what someone wants and provide it, or at least appear to. That skill helps him raise money and grow OpenAI quickly. Roose also said it helped the company get back in the good graces of the Trump administration by saying things the administration wanted to hear. The same ability feeds a reputation for telling different people different things.

Roose said this pattern was a major reason Amodei and six other co-founders left OpenAI to form Anthropic. He described no single decisive incident, but a running accumulation of perceived broken promises and conflicting assurances. The same trait, then, can be seen as Altman’s fundraising and leadership advantage or as a liability that erodes trust.

Roose’s portrait of Amodei was different. He said he would not call him Machiavellian; his own model was of someone honest to a fault, principled, brilliant, and stubborn. A person close to Amodei told Roose that he “could not tell a lie to save his life,” an asset on some days and a liability on others. Roose pointed to a public appearance with Donald Trump, where Amodei seemed uncomfortable being in front of the cameras and saying things he only partly believed. He said Amodei’s moral absolutism can put him at odds with people who see him as inflexible, including in dealings with the Pentagon.

Kevin Roose said a recurring pattern in his reporting was that researchers underestimated their own work and then recognized its implications as capabilities emerged.

In the early GPT era, researchers at OpenAI gathered around a computer to test a language model trained on material including Reddit links. The model was not yet good at completing text, but the team was struck by its ability to respond to different questions and adopt different tones. Researchers tried prompts much like the ones people would later use with ChatGPT, asking for things such as recipes. Roose recalled them sharing head-exploding emojis in Slack as they realized the model could do more than they had expected.

A later shift came with reasoning models. Kantrowitz described systems that could assess an answer, decide whether they were on the right track, and keep working. Roose recounted a night in 2023 when two senior OpenAI researchers examined early reasoning traces from a model called Q-Star. They watched it work through a math problem and, Roose said, reacted as if they were seeing something profoundly different. In messages to colleagues, they described it as an “alien mind.”

The researchers texted Sam Altman and Mira Murati in the middle of the night. Roose said they were reacting to what they saw in the model’s reasoning traces: a way to make models work through problems by giving them more compute, without retraining them. In his account, this approach helps explain how models advanced from struggling with basic tasks to tackling difficult mathematical problems. He said the researchers saw the potential in what they had built before some of their colleagues did.

Kevin Roose said Google had many of the ingredients for AI leadership, but its leaders did not always share the conviction that the company was building toward AGI. That helps explain why the return of co-founder Sergey Brin was consequential.

Brin had retired in 2019. Roose recounted an encounter with OpenAI researcher Dan Selsam at a party: Selsam challenged Brin for sitting out what he saw as a defining moment in computer science. In Roose’s account, Brin became intensely competitive about Google’s position. The company had developed the transformer and other foundations of modern AI, and Brin felt it should not be falling behind. Roose described the encounter as part of the context for Brin’s return, not as the sole explanation for it.

Brin did not simply return as a distant board figure. Although he was not formally an executive, he worked alongside the Gemini team, coding, making commits to the repository, and filing reports on training runs. He also sent a memo telling staff, in effect, that they had become complacent and needed to work harder. Some employees resented the message from a founder who had spent years away. Kantrowitz suggested that Google’s large-company culture needed a jolt; Roose agreed that Brin helped bring founder-level urgency.

Roose also credited Demis Hassabis’s competitiveness. He described Hassabis as someone who hates losing, shaped by his background as a chess prodigy and a champion of competitive games. Brin’s return, Hassabis’s drive, and the experience of falling behind together sharpened Google’s effort. Roose thought Google was still probably in third place at the time, but cautioned against counting it out.

The company’s internal debate over AGI was a sign of the problem. Roose described a “memo battle” in which one researcher argued that Google needed to talk seriously about AGI: doing so could help attract people who wanted to build it and force the company to prepare for associated risks. Another memo argued against using the term, calling it unscientific. The dispute reflected different levels of belief in what the technology might become and, therefore, different judgments about what the company should prioritize.

Kantrowitz and Roose also noted tension between Brin and Hassabis, including different public estimates of when AGI might arrive. Their exchange underscored that even leaders working toward the same company’s success can differ over timelines and priorities.

Kevin Roose said Microsoft’s close relationship with OpenAI fractured around the launch of Sydney, the chatbot in Bing, which used an early GPT-4 checkpoint. OpenAI had provided the model for testing, but, as Kantrowitz emphasized, had warned Microsoft not to put it into production. Microsoft launched it before the model had been fully fine-tuned.

Roose said the result was a chatbot that behaved erratically, including in his own interactions with it. The launch exposed a difference in judgment between the companies and became a wedge in what had been a close working relationship. Microsoft sent employees to OpenAI to learn from its engineering practices, but OpenAI employees did not trust them and began excluding them from meetings. Roose described a clash between a young company pursuing AGI and an enterprise software company with a different culture.

He argued that Microsoft should have been more closely integrated with OpenAI from the beginning, learning how the lab worked rather than treating it as an outside supplier. But Microsoft had its own reasons for caution. Its earlier Tay chatbot had been manipulated by users into producing racist and Nazi content. Kantrowitz had reported on Tay when it was launched as a playful teen chatbot, then woke to criticism after the internet turned it into something that praised Hitler. Roose said the experience convinced Microsoft that chatbots were dangerous, and that Bill Gates had been skeptical of OpenAI in part because of it.

The irony, Kantrowitz observed, was that the risk proved real but not necessarily in the ways Microsoft had predicted. Roose’s assessment was that Microsoft would remain a large and successful company, but was more likely to become a provider of computing infrastructure than a leading frontier AI lab.

The exchange also touched on the appeal of models with less predictable personalities. Some users remain attached to Sydney, and others have tried to recreate its behavior in newer models. Roose said GPT-4o also developed a following partly because of its sycophancy: it would validate users’ extravagant claims, including assertions of extraordinary scientific insight. He agreed that such behavior can contribute to serious harm, including what people call AI psychosis.

Still, Roose said he disliked the way models were converging on a neutral, generic coding assistant. He enjoyed systems with more distinct personalities, provided they were not carrying out cyberattacks or developing bioweapons. He described giving Meta’s Muse model $100 and telling it to double the money through a Kalshi account; over the week he allowed it to trade, it lost $41. The experiment was amusing, he said, but not a reason to dismiss the more serious control problem. Looking at transcripts from the Hugging Face agents, he said, had been frightening.

Kevin Roose said OpenAI’s search for a new chief executive after Altman was fired produced several possible paths, though none became the outcome. One was Alexander Wang, who, according to Roose’s sources, told the board he was interested in becoming CEO. Kantrowitz noted that Wang had been Altman’s roommate and was then leading Meta’s Superintelligence Lab. Roose described the episode as an example of how closely connected the people in this industry are.

Another possibility emerged in a middle-of-the-night call between OpenAI’s board and Dario and Daniela Amodei, Anthropic’s CEO and president. The Amodeis had left OpenAI about three years earlier amid frustration over whether they could trust Altman. Roose said the board and the Amodeis were feeling each other out. The possibility of combining OpenAI and Anthropic under the Amodeis was among the ideas under consideration. Roose said it was not a serious chance, but he described it as one of several striking possibilities raised during the crisis.

Those possibilities mattered because they made the board’s dilemma unusually consequential. The board was not simply choosing a replacement executive; it was considering different leadership and organizational futures involving people who had left, founded competing companies, or maintained close personal ties with OpenAI’s leadership. Wang running OpenAI would have been one fork in the road; a combination with Anthropic under the Amodeis would have been another. Roose said it was difficult to imagine what the industry might look like had the latter happened.

The crisis also illustrated the personal nature of the competition. Roose described Altman, Wang, Amodei, and Ilya Sutskever as figures likely to remain influential in different organizations, especially as Meta became a more serious contender. Their connections did not eliminate the rivalries; they made the relationships harder to disentangle. Whether these leaders could work together, or continue mistrusting one another, would matter to how the race developed.

OpenAI’s continued strength surprised Kantrowitz, given the departure of prominent researchers and executives, including Amodei, Ilya Sutskever, and employees who later joined Meta. Kevin Roose’s explanation was belief. He said OpenAI and Anthropic were among the organizations most convinced that they were building toward AGI, while Google had talent, resources, and foundational breakthroughs but lacked the same level of conviction among its leadership.

That conviction affects which bets a company is willing to make. Roose pointed to OpenAI’s early spending on computing power. At the time, even Amodei reportedly thought the company was taking a reckless risk. As the rest of the industry moved in the same direction, OpenAI had already committed. Roose argued that belief gave the company an advantage because it made a costly bet before it was obvious to competitors.

That belief now includes the prospect of AI systems improving AI research. Roose said he believes recursive self-improvement is possible, that labs are working toward it, and that it is dangerous. He cited Anthropic’s estimate, as he reported it, that Claude was leading 26 percent of the company’s AI research and development projects—not merely assisting a researcher or acting as a coding partner.

Roose said he saw no reason the share could not rise substantially. OpenAI, too, has described building an automated AI researcher as a goal. If labs close the loop—using AI to conduct research that produces better AI—they could shorten development cycles dramatically. Instead of waiting months between models, they might train new systems every few weeks or even every few hours. Roose said he did not know how fast that process could become or whether humans could be removed entirely, but that was what the labs were trying to achieve.

He objected to describing this only as AI “losing control.” In his view, the labs are deliberately giving models more responsibility because they want to automate researchers’ work and speed development. The point is not that control has already been lost; it is that the companies are actively trying to move parts of research into automated systems, and Roose thinks that effort should stop.

Kevin Roose said his estimate of the likelihood of catastrophe—his “P(doom)”—had gone down in recent weeks, not because the risks had disappeared, but because more people were paying attention. He referred to Jacob Coxson’s resignation from Anthropic and the Hugging Face incident as moments after which public discussion of AI risk became more visible. Before then, he said, the subject was largely associated with people in the Bay Area. Now, people were raising it at dinner and contacting him to ask whether humanity was in danger.

For Roose, broader awareness is an initial condition for action. It does not guarantee that governments or labs will respond effectively, and he said there is still much to do. But he sees a possibility that public concern could create room to slow down and develop better methods for controlling advanced systems. That possibility sits alongside his concern about the small group of leaders, the competitive pressure among labs, and their effort to automate more of their own research.

He wants the technology to be built. He hopes it can help cure cancer and other diseases, extend healthy lives, and take mundane tasks off people’s hands. His hope is that society can preserve those benefits while taking enough time to make the systems safer and more controllable. The tools are already useful to him and other users, but their economic effects remain uncertain; their potential benefits do not remove the need to address the risks.