AI Safety Rules Could Concentrate Control Over Frontier Models

Jacob Coxon’s resignation from Anthropic turned a familiar argument inside frontier AI labs into a public political event. Coxon wrote that people building AI “earnestly believe that it could kill us all by the end of the decade,” and said OpenAI and Anthropic were “racing straight to self-improving superintelligence and gambling with our lives.”

The claim gained force when Evan Hubinger, described in the discussion as leading alignment science at Anthropic, publicly agreed with Coxon’s premise. Hubinger said he personally put the chance of AI killing all humans within a decade above 10%, and that Anthropic did not yet have a plan to solve alignment for superintelligence or appear clearly on track to do so.

The posts quickly reached political figures. Bernie Sanders said Coxon was right and said he would introduce legislation to ban superintelligence and pause AI development. Illinois Governor JB Pritzker called for immediate action by industry and Washington. The speed with which a resignation post moved into national political rhetoric was itself part of what the hosts objected to.

David Sacks rejected Coxon’s designation as a whistleblower. In Sacks’s framing, a whistleblower would ordinarily bring documents, reports, technical evidence, or facts previously unavailable to the public. Coxon had instead offered an alarming conclusion without supporting material. “Show us the data, show us the reports, show us the leaked information,” Sacks said. “Show us the facts, show us the evidence.”

Sacks also argued that the rollout was organized rather than spontaneous. Coxon’s account had almost no visible activity, followers, or prior posts before the resignation thread, he said, yet it rapidly reached a vast audience. Sacks pointed to early amplification by people associated with Encode AI, the AI Policy Institute, and the AI Futures Project. He also noted that a Wall Street Journal article about Coxon’s resignation appeared to have been published minutes before Coxon’s own post. Jason Calacanis said the more mundane explanation was that the paper had been briefed under embargo and published at the wrong time.

Sacks described overlapping relationships among those organizations and Jaan Tallinn, whom he identified as both an effective-altruist donor supporting AI-safety efforts and a co-lead of Anthropic’s Series A. He later added Dustin Moskovitz and Sam Bankman-Fried to his account of Anthropic’s early funding. His claim was not that every person responding to Coxon belonged to one coordinated conspiracy. Rather, he argued that a deliberate amplification campaign could emerge among people who also sincerely hold catastrophic-risk views.

Calacanis separated the possibilities more explicitly. The researchers could be right because they had seen something outsiders had not; they could sincerely believe a mistaken theory; or they could be participating, knowingly or not, in a campaign whose practical result would be tighter regulation and less open-source AI. Calacanis said he leaned toward the view that many people advancing the warning genuinely believed it but were wrong.

The hosts did not deny that AI could create serious harms. They accepted that AI can be used in cyberattacks, biological work, military systems, and other high-consequence domains. Their objection was to the implied route from current capability gains to autonomous human extinction.

Sacks said the strongest version of the danger case is recursive self-improvementA proposed process in which an AI system repeatedly improves its own capabilities through successive generations., or RSI. The feared sequence is that an AI becomes capable of doing the work of an AI researcher; it designs, initiates, and evaluates a training run for a stronger successor; and that successor repeats the cycle without needing human approval. Improvements compound, and people lose the practical ability to direct or stop the system.

Sacks distinguished that possibility from what he called “prosaic” recursive self-improvement: researchers using AI to write code, analyze experiments, or speed up pieces of model development while people still decide what gets trained and deployed. The consequential claim is RSI maximalism—an AI devising its own next training run, starting it, and repeating the process without a human in the loop.

David Friedberg argued that the distance between those conditions remains substantial. Current models can coordinate agents and accomplish pieces of complicated digital work, but they still encounter physical and organizational bottlenecks. An AI might navigate the digital steps required to request a toiletry kit at a hotel, Friedberg said, but a person still has to retrieve the item and carry it to the room. The same human dependencies that limit end-to-end productivity also constrain an AI’s ability to act through critical systems.

Friedberg pointed to financial networks as his strongest imaginable route to broad instability, but said institutions maintain physical records, distributed backups, recovery procedures, and deliberately disconnected systems. Some infrastructure is air-gappedSeparated from internet-connected networks to reduce exposure to remote compromise.; other processes require human approval. Digital compromise, in his view, does not establish that every redundancy and physical control would fail at once.

Calacanis and Chamath Palihapitiya nonetheless tried to steelman catastrophic scenarios. Calacanis proposed an AI-enabled military system that helps trigger nuclear conflict. Palihapitiya imagined an AI gaining access to internet-connected robots, bioreactors, precursor materials, and the ability to produce and disperse a lethal airborne pathogen.

Palihapitiya’s own point was that such a scenario requires a long chain of conditions: a sufficiently capable model, access to specialized equipment and materials, remote control over that equipment, a viable pathogen, global distribution, and failed countermeasures. He called the scenario implausible because so many conditions would need to hold simultaneously.

Sacks’s preferred way to assess such claims was to identify those intermediate conditions rather than jump from current AI progress to extinction. A Bill Gurley post shown during the discussion recommended the “Five Whys” technique: identify what must happen next, then keep examining the technical, institutional, and physical steps required to reach catastrophe.

The hosts’ answer to RSI maximalism was not that it should be ignored. Sacks said it was something to watch. But they argued that developers can impose approval gates before consequential actions. Current agents already ask permission before accessing email, submitting forms, or completing transactions. In the hosts’ view, it is not inevitable that a lab would give a model authority to launch successive training runs without an accountable human decision.

That is the underlying empirical dispute. Coxon and Hubinger’s public statements treat uncontrolled self-improvement as sufficiently near and consequential to justify urgent action. The hosts treated it as a speculative endpoint whose intervening steps have not been demonstrated.

The governance question follows from the technical one, but it is not the same question. Even if AI risks justify testing, monitoring, and limits on particular uses, the hosts argued that the form of those controls will determine who is able to build and operate models.

David Friedberg supported safety work in principle. AI systems should be tested, developers should build safety mechanisms, and companies should respond thoughtfully to genuine emerging risks, he said. His concern was a regime that makes centralized control a condition of legality and gives a regulator continuing authority to decide when development may proceed.

Open-weight models are central to that concern. Friedberg described open source as a way to lower the cost of AI, allow people to run models on their own devices or infrastructure, and prevent the gains from accumulating entirely to a small set of proprietary cloud providers. In his account, open source means that the technology can be used and developed more broadly rather than solely through a frontier lab’s API.

That distribution model conflicts with regulatory standards based on recall and centralized monitoring. David Sacks said Dario Amodei had argued in Senate testimony that models can be dangerous when they cannot be centrally monitored, controlled, and rolled back. Once model weights are publicly released, Sacks argued, they cannot be meaningfully recalled: they can be copied, modified, hosted elsewhere, and used outside the original developer’s control.

Sacks’s concern was that a regulator could apply the same monitoring, safety-review, and rollback requirements to open and closed models. A hosted provider may have staff, compliance systems, update channels, and control over its own service. Open-weight developers and users may be structurally unable to provide equivalent control. That could lead, he argued, to restrictions on publishing, hosting, or using weights that fail a centrally administered standard, even if the rule is not formally described as an open-source ban.

Friedberg’s objection was that this would make a regulator the keeper of the “gas pedal.” Someone would decide which systems could operate, which developers could satisfy the compliance process, and when capabilities could advance. In his view, centralized control could create monopoly or oligopoly conditions even if it is introduced in the name of safety.

Sacks put the political economy more harshly. AI-safety groups seek more regulatory authority, he argued; politicians gain a new domain of state power; and large frontier providers benefit if compliance costs exclude smaller firms and open projects. In that account, existential-risk rhetoric can align ideological, political, and commercial interests even when every participant is not part of a single plan.

Friedberg also objected to a U.S.-only development pause on strategic grounds. A recursively improving system, he argued, would require adequate chips, energy, and connectivity—not permission from the U.S. government. If a domestic ban works only inside the United States, a foreign company, government, or other organization could still develop the capability and gain an advantage.

He said technology discussed ahead of the group’s summit was reducing power cost per output token by as much as 10,000 times, lowering the barrier to experimentation. His point was not that every individual could easily train a frontier model. It was that a national prohibition would not necessarily control a globally distributed technical capability.

Sacks extended the concern from model access to speech. He imagined a federal AI regulator pressuring providers to classify disputed views as disinformation and shaping the answers users receive from personal AI systems. Drawing on his interpretation of COVID-era content moderation, Sacks argued that users asking an AI about medical risks could receive only an officially sanctioned answer if providers were required to prevent disfavored views from appearing in their systems.

The hosts therefore treated the practical policy question as one of proportionality. Their preferred alternative was not no safety work, but controls directed at demonstrable harmful uses rather than a licensing-and-monitoring system that, in their view, only centrally controlled models could realistically survive.

Anthropic’s anticipated IPO made the safety argument more than a general policy dispute. A Polymarket screen shown during the discussion put the chance of an Anthropic IPO before 2027 at 88%.

The narrow issue identified by the hosts was not Coxon’s resignation by itself. It was Hubinger’s public endorsement. A company may be able to characterize a recently departed employee’s claims as mistaken, disgruntled, or self-promotional. But Hubinger was described as a current senior alignment executive managing a safety team, and he wrote that he personally saw a greater than 10% chance of AI killing all humans within a decade.

Palihapitiya argued that this creates a difficult disclosure question. During an IPO filing process, he said, the company and Securities and Exchange Commission pressure-test the S-1 as a current account of the company’s opportunities and risks. Public investors use that document as an anchoring description of the business they are being asked to finance.

He recalled that public comments had complicated other IPO processes. An interview with Google’s founders published during its quiet period created concern about whether the company might need to refile. Palihapitiya said comments he made about Slack’s network effects while Slack was going public nearly required corrective treatment in that company’s filing.

Anthropic’s situation, in his telling, is more difficult because the risk concerns the company’s core product. Typical risk factors cover uncertainty around demand, competition, supply, regulation, and market conditions. A current safety executive publicly saying that the company has not solved alignment for a potentially civilization-ending system is not ordinary boilerplate, Palihapitiya argued.

He said investors could ask whether the statement is material and whether it belongs in the S-1. They could also ask what liability follows if a company’s own personnel describe its product as carrying grave foreseeable risks. Palihapitiya believed that uncertainty could demand a large valuation discount, although he did not claim to know what investors or regulators would ultimately do.

David Sacks said his post calling for Anthropic’s IPO to be paused was tongue-in-cheek, signaled by quotation marks around “whistleblower.” But he argued that Anthropic faces a real strategic tension. If it rejects Coxon’s claims as unsupported hyperbole, it risks conflict with employees who share his outlook. If it effectively validates the claims, it becomes harder, in Sacks’s view, to explain why it should continue developing frontier systems or why public investors should fund the company at a very large valuation.

Palihapitiya thought Anthropic could be caught between commercially minded employees building a capital-intensive business and a sufficiently large internal group committed to catastrophic-risk reasoning. He argued that a disclosure may not eliminate the consequences if customers, investors, or future claimants conclude that the company knew of a serious danger.

Calacanis focused on corporate discipline. Companies approaching an IPO generally impose communications policies and tell employees not to make public statements about material issues, he said. He asked why Anthropic insiders were publicly co-signing statements with direct implications for the company’s products and valuation.

Palihapitiya cautioned that outsiders do not know Anthropic’s internal instructions. The important public fact, as he saw it, was the pile-on: regardless of what management had said privately, a current safety executive had associated the company with the extinction-risk claim.

Sacks contrasted Anthropic’s response with remarks attributed on screen to Nvidia CEO Jensen Huang, who called Coxon’s comments “outlandish” and “deeply untrue” while praising safety work around the industry. Sacks’s question was why Anthropic could not be equally clear. His answer was that the company’s safety culture and regulatory strategy made an outright repudiation difficult.

The hosts did not establish what the SEC must require, whether a quiet-period violation occurred, or what future litigation would produce. Their argument was narrower: a current executive’s statement can matter more than an ex-employee’s allegation because it raises questions about what the company believes, what it has disclosed, and whether its external messaging is internally coherent.

OpenAI’s claimed solution to the Navier–Stokes Millennium Prize Problem raised two separate questions: what AI systems can accomplish through large-scale computational coordination, and whether customers can trust frontier providers with unpublished work.

OpenAI said its proof was produced by a group of agents using a next-generation model more capable than GPT-6 Astra. The company said the problem concerns whether smooth three-dimensional fluid motion modeled by the Navier–Stokes equations can break down.

David Friedberg treated the announcement primarily as evidence of computational leverage rather than machine mysticism. OpenAI reportedly used 10,000 agents and 130 billion output tokens, he said. In his description, those agents were parallel software processes exchanging information and analysis, not independent minds having an incomprehensible flash of genius.

Friedberg estimated that the work represented somewhere between 50 and 500,000 years of human effort, depending on how token output is translated into human labor. Even after substantial adjustment, he said, the system had compressed an enormous amount of intellectual work into a short period.

For Friedberg, the practical implication is that AI can apply known techniques at radically different scale. Aircraft wings, engines, energy systems, and other complex designs may become candidates for much faster computational search. The work remains inspectable, he argued: people can read the exchanges between agents and assess the reasoning that produced an answer.

The controversy arose because mathematicians Levent Alpöge and Tristan Buckmaster had been working on related questions while using OpenAI products. OpenAI congratulated them and said neither its researchers nor agents had seen their work before it was publicly released. It specifically denied accessing user data to solve the problem. But it also said it could not rule out that de-identified data from the researchers’ product use had helped improve its models.

That distinction became the center of the discussion. Directly accessing identifiable prompt histories would be one kind of allegation. A provider learning from de-identified interactions used to improve a system would be another.

David Sacks said he found direct prompt access implausible. He cited OpenAI researcher Noam Brown’s statement that nobody had looked at Alpöge and Buckmaster’s prompts and that doing so would be “insane.” Sacks’s own view was that OpenAI had heard researchers were making progress on Navier–Stokes and devoted substantial compute to the problem. Calacanis added that Sam Altman had reportedly said OpenAI heard Anthropic was getting close and decided to take a swing.

Sacks gave OpenAI the benefit of the doubt on the narrower allegation of people rummaging through identifiable prompt chains. But he accepted that the broader question of data privacy and model learning is serious.

Friedberg argued that de-identification does not settle the intellectual-property issue. A person’s name and employer can be removed from a record while the valuable content remains. A mathematical approach, scientific hypothesis, technical design, or business strategy may be precisely what a customer is attempting to protect.

Friedberg said he had seen what he regarded as this pattern in his own scientific work. After asking a model about a novel idea, he later queried another account or a later model version and received an answer that appeared to incorporate the earlier insight. He acknowledged these as anecdotes rather than proof. But he said he knew the narrow domain well enough to doubt that the information had entered a new public corpus in the interim.

The issue, in Friedberg’s framing, is the closed-model network effect. A provider can observe the problems users are working on, the approaches nearly succeeding, and the methods people use to make progress. That information may improve the provider’s model even after identities are removed. The platform’s broad visibility becomes a competitive advantage; the customer may have supplied an insight that strengthens it.

Chamath Palihapitiya described zero data retention, or ZDR, as a commercial best-efforts promise rather than an absolute guarantee. A user may ask a system not to retain the process behind a protein design, he said, but feedback mechanisms, logging, training pathways, or other product behavior can create routes by which useful information enters a broader corpus.

His proposed answer was a sovereign deployment: infrastructure, hardware, and model access arranged so the enterprise controls the environment and the data path. He named AWS, Nebius, and CoreWeave as potential infrastructure providers. The model could be open source, but that was not the only option; Palihapitiya said a customer might also use a controlled version of a closed model in a virtual private environment.

Calacanis cited movement in that direction. Harvey had announced its proprietary legal AI model, Tenet, based on Qwen K3. Calacanis also promoted Go.ai, an incubated company building on-premises AI infrastructure for regulated industries, and said he was moving much of his own sensitive work away from Claude toward local open models.

Palihapitiya cautioned that a local machine is not a complete enterprise answer. Large organizations require multi-user collaboration, memory, knowledge bases, and scalable access. His concern was not simply where a machine sits, but who controls the full architecture, including opaque internal model processes that users cannot inspect.

He argued that risk and audit committees will increasingly ask CIOs to explain exactly what has been exposed through hosted-model use. If a company later cannot establish whether proprietary information leaked into a model or contributed to a competitor’s result, he predicted that shareholders and boards will demand accountability.

Sacks raised a separate consumer privacy concern. He said personal AI chats may not have protections equivalent to email. In many contexts, he argued, government access to email requires a warrant and probable cause, whereas AI-chat data can be obtained through a subpoena or court order. That distinction matters as people use AI systems for legal, medical, therapeutic, and deeply personal questions.

His proposed baseline was stronger privacy law. If users ask an AI a question they would otherwise ask a lawyer, doctor, or confidant, Sacks argued, the legal protection should reflect the intimacy of that interaction. The enterprise problem requires contracts and deployment controls; the consumer problem requires clearer limits on access to chat data.

The concern grows when model providers compete with their customers. Sacks pointed to frontier companies releasing coding and design products that can compete with businesses built on their APIs. If a platform can learn from customer use and enter adjacent application markets, he argued, customers need more than general assurances about responsible handling of their data.

Nike’s removal from the S&P 100 after 18 years in the index gave the hosts a separate example of strategic drift. Calacanis cited a peak market capitalization of $264 billion in 2021, peak revenue of $51 billion in 2024, and a share-price decline of roughly 80% from its peak. Palo Alto Networks was replacing Nike in the index.

The hosts’ diagnosis combined commercial execution, product experience, China exposure, and political branding.

Calacanis pointed first to decisions under John Donahoe, who became CEO in 2020. Nike pursued a more aggressive direct-to-consumer strategy, he said, reducing the role of retail partners that had helped build the brand through shelf space, local expertise, and customer relationships. That opened room for competitors such as Hoka and On. He also cited a 30% China-sales decline, eight consecutive quarters of decline, and share losses to Chinese brands Anta and Li-Ning.

David Sacks emphasized Nike’s branding choices. He argued that Nike had historically stood for athletic performance, victory, and elite achievement, built through figures such as Michael Jordan. In his view, campaigns involving Colin Kaepernick, Dylan Mulvaney, and non-athlete models displaced that identity with political messaging he described as “woke.”

For Sacks, the broader failure was Nike’s departure from the athletic aspiration on which it had built the brand. He also criticized Nike’s internal reorganization. The company had previously structured parts of its business around sports such as basketball, football, tennis, and swimming, he said. Moving instead toward broad demographic categories such as men, women, and children appeared to him to weaken the relationship between product teams and particular athletic disciplines.

Chamath Palihapitiya framed the diagnosis around Nike’s lost North Star: “mastery and excellence embodied through athletics.” Nike’s strongest associations, he said, came from consumers seeing Michael Jordan, Tiger Woods, Serena Williams, Pete Sampras, and other exceptional athletes and wanting to participate in that aspiration.

The product did not need to make a consumer identical to the athlete. It gave people a connection to the version of themselves they hoped to become. Palihapitiya argued that Nike should encourage customers to improve themselves and return to sports, rather than move its central message away from excellence.

Friedberg supplied a product-level explanation from his own experience. He said he had once bought only Nike shoes, but began finding they fell apart in roughly six weeks and no longer had the durability he expected. He moved to Brooks after trying a pair at REI and found them more comfortable and longer lasting for running.

Friedberg cited Brooks as a counterexample: a Berkshire Hathaway subsidiary that, he said, had grown revenue at double-digit rates for nine consecutive years and reached $1.6 billion in revenue. He attributed its operating principle to an instruction from Warren Buffett: make the product better every year. In Friedberg’s account, that product discipline contrasts with Nike’s perceived shift toward narrative.

Palihapitiya argued that Nike still retains substantial latent strength. He said he would return as a customer if the company restored its association with athletic mastery, rebuilt a retail presence where consumers have disposable income, and made products people actively wanted to wear.

The hosts’ shared point was that a sports brand cannot rely on distribution strategy or social messaging alone. It has to make better products and preserve a credible reason for customers to aspire to the people and performance it represents.