Cambridge AI Safety Proposal Calls for Shared RSI Measures and Crisis Planning

The Saturday Night Live parody of Anthropic CEO Dario Amodei worked, John Coogan argued, because Amodei has become recognizable beyond the technology industry. His appearances on major television networks have made him familiar enough for a national comedy show to build a character around him. Jordi Hays noted that even Amodei’s sweater vest had become part of the impression.

The sketch’s joke was not simply that Amodei talks about AI risk. Its character seemed unable to describe the danger without asking lawmakers to stop him, and unable to offer reassurance without returning to the risks. One line condensed the tension: “AI is not a weapon. It’s a tool. A tool for building weapons.” The sketch made the contradiction legible to a broad audience: a company leader warning that the technology he is building could be dangerous while arguing for guardrails to reduce that danger.

Hays said Amodei is fascinating to listen to and unusually clear in his thinking, but argued that he could improve how his warnings land. In particular, Hays said, he should stop smiling while discussing devastating scenarios. Coogan compared the challenge to the public manner of a Fed chair, FDA head or military leader: seriousness can be a performance, but it communicates that the speaker understands the stakes. He contrasted that with Mark Zuckerberg’s more upbeat posture toward AI, which may appeal to people looking for a more optimistic message but, Hays suggested, is not necessarily appealing to researchers at the frontier.

The public debate was also becoming harder to separate from politics. After the sketch, Amodei was due to have a private dinner with President Donald Trump. Coogan described the dinner as a chance to repair relations—or make matters worse—and said Trump had invited Amodei after the Anthropic chief appeared to be left out of another White House dinner involving technology leaders. Coogan also pointed to another planned meeting in Washington with AI leaders, saying the pace of political engagement was picking up. The hosts joked about possible outcomes, but the meetings underscored how lab leaders and elected officials were increasingly being drawn into the same argument over AI.

Against that backdrop, Coogan described a University of Cambridge proposal as more concrete than the familiar suggestion that AI companies simply talk to one another. As he recounted it, one proposed step is to establish comparable measures for how much AI contributes to research and development: what share of the code, work or spending is being handled by AI systems. Companies have described growing AI contributions, but Coogan said their measures differ, making it difficult to compare how far along different labs are. He also noted that labs may have financial incentives to appear further along the curve of recursive self-improvement, or RSI.

Coogan said the proposal would also ask governments to prepare responses to severe failures rather than design them during a crisis. He gave the example of an AI agent causing a major cyberattack or disrupting internet infrastructure. Governments have plans for responding to disasters such as hurricanes, he said; the proposal asks what the equivalent would be if an agent brought down critical systems. Biosecurity and economic disruption were also among the areas where preparation was needed. As Coogan described it, the proposal identifies areas for government readiness; it does not supply finished response plans.

Hays wondered whether people might one day rehearse AI-disaster scenarios in simulations, as they might practice for other emergencies. The exchange turned playful, but returned to the practical question raised by the proposal: what would institutions measure, and who would be expected to act if a crisis occurred?

The prospect of AI failure also appears in private preparations. Coogan cited a report that some early Anthropic employees were considering buying remote land in the United States as a place to relocate if AI went seriously wrong. The report also described employees exchanging doomsday scenarios and preparation plans in a private Slack, and referenced the wider safety community’s discussions of iodine pills, remote islands and shielded desert bunkers.

Coogan connected buying land to a question often asked of people who take AI risk seriously: if they believe catastrophe is possible, why not spend money preparing? Hays pressed on the practical weakness of a remote retreat. Someone arriving in a community where they have no relationships may not be welcomed, especially if others know that the newcomers have stocked a well-equipped shelter.

Coogan supplied the prepper-community term for the problem: “Don’t be a loot drop.” A bunker full of supplies can become a prize for other people if its owners are not integrated into the community around it. The exchange made the social dimension of survival planning explicit: a secure location does not, by itself, secure cooperation from the people nearby.

The business case for personal agents came into view through Instinct, whose founder Noah Shinn discussed the company with Patrick O’Shaughnessy. The hosts reviewed figures from that interview and its accompanying on-screen graphics, but the headline transaction-volume figure was ambiguous. Coogan was unsure whether the reported $1 billion meant total volume to date or an annualized rate; Hays had read it as annualized. Hays compared it with Stripe reaching $1 billion in annualized transaction volume after about 18 months, while noting that Stripe had spent two years building in beta and that Instinct had reached its reported figure in closer to six months.

Other figures attributed to the interview or on-screen summary included growth above 10% a day, no marketing spend, and compute demand roughly doubling every week. The product was still invite-only. The figures also said that 40% of users shared a personal credit card with Instinct within three weeks, about half of its transaction volume was travel, and some small businesses ran their entire back office on it. Users who connected at least one piece of sensitive information reportedly retained at about 80%. The interview summary described the company’s own A/B tests as matching Opus 5 performance at a fraction of the cost.

The hosts discussed screenshots of the product, including examples of mistakes, as a possible source of its visibility. Hays wondered whether those failures might increase interest by making people curious about the product. Coogan thought they could. Hays added that many people in technology have already experienced AI making something for them, while most people outside the industry have not; the first experience of having a system act on one’s behalf can feel magical.

The reported usage points beyond chat. Some users sent more than 90% of their messages by voice, and the interview summary said Instinct had proactively called Shinn three times when a task had a time-sensitive deadline. The on-screen figures also said background work could run on setups three, five or eight times more efficiently on the same compute. Coogan noted that Shinn spent about 40% of his time on compute, and that buying compute at the last minute cost three to four times more. In the figures the hosts discussed, demand was rising alongside the work of securing and managing compute to serve it.

Monetization was another open question. Instinct planned to take a cut of purchases made through the product, with the hosts comparing possible rates to Shopify, Amazon and Apple. The transactions already flowed through third-party cards, so Hays said Instinct was not yet earning money from them. He suggested that a company card could create a path to interchange revenue, even at a small fraction of transaction value. The product already had, in the hosts’ description, a computer, phone number and email; adding a payment method could make purchasing more direct.

The founders’ ambition, as relayed by Coogan, was to keep the product free for everyone for life. Another part of the business case was that agents could replace advertising-driven discovery while increasing purchases: if agents remove the effort involved in buying, users may buy more. Coogan questioned how much additional convenience that would create when online checkout is already quick. Hays offered a different use case: shopping for low-stakes, undifferentiated things—such as cable-management supplies—where a user wants a good solution, not a brand-searching exercise.

The on-screen figures said the company had launched to about 200 friends and family, its invites had sold on eBay for around $300, and its team numbered 14. They also described it as having raised $1 billion at a $10 billion valuation less than a month after raising at a $2.5 billion valuation. Hays’s investment framing was that the scale of transaction volume and possible future monetization made the valuation understandable if growth continued, while acknowledging that the company would have to keep scaling and withstand competition. He also pointed to potential partnerships, including with Amazon, as an avenue the company might pursue.

The ability to make decisions on behalf of users raises a systemic concern. John Coogan read a warning that widespread use of personal-finance agents could trigger a bank run or market flash crash if many systems independently made similar decisions. The argument was that agents do not need to communicate or collude to behave in a coordinated way: if they optimize for the same goals using similar information, they may arrive at the same action at roughly the same time.

Jordi Hays related that concern to the broader idea that parts of the economy depend on friction. An always-on agent might notice unused subscriptions or move idle cash into a higher-yield account. But Hays resisted treating ordinary users getting better financial outcomes as an economic disaster. A strong banking service already handles some of these tasks proactively, he argued; the change would matter more for customers who are not receiving that kind of service.

Coogan’s caution was about adoption and trust. Personal agents were still being used by a relatively small number of people, he said, and broad deployment would require users to authenticate accounts, link financial information and decide how much authority to give the system. Early versions might ask for permission before acting; a later, more trusted agent might simply do something it knows the user wants. Building that trust and spreading the practice could take years.

He compared the expected pace with early claims that ChatGPT would quickly make Google Search obsolete. ChatGPT was growing rapidly, he said, but the shift did not happen overnight. The same could be true of agents: if adoption takes several years, banks and financial products may have time to adjust. Coogan said the risks were worth monitoring, while emphasizing that adoption and trust would take time.

Meta’s new Enterprise Platform brings its models, agents and infrastructure into a business-facing unit, according to Mark Zuckerberg’s announcement. The initial offering would include the Muse agent, Meta Business Agent, Muse API and Muse Code for businesses and developers. Zuckerberg also named former MongoDB CEO CJ Desai as chief enterprise platform officer, reporting directly to him.

Jordi Hays interpreted the new unit as a way to organize a large enterprise business, potentially including revenue from substantial compute deals. He speculated that the unit might quickly be described as having grown from zero to billions in annual revenue, without specifying how much came from software, services or infrastructure. John Coogan said the revenue could take several forms: tokens, inference, raw compute, bare-metal deals or services. The announcement established Meta’s ambition to sell a broader technology stack to businesses; where the revenue would come from remained open.