AI Fatalism Is Blocking Real Choices on Regulation and War

Brad Carson rejects the idea that AI development has already escaped political control. His claim is not that every restraint is wise, or that every international agreement is practical. It is that fatalism narrows the policy imagination before the real choices have been debated.

People often say of AI, Carson argues, that “it’s coming” and society simply has to accept it. He calls that view false and dangerous. Governments and scientific communities have redirected powerful technologies before. In war, states have agreed not to use weapons or tactics that might be militarily useful: biological weapons, chemical weapons, dum-dum bullets, and the killing of surrendered or wounded soldiers. Nuclear strategy provides the darker example. After the Cuban Missile Crisis, the United States and Soviet Union did not conclude that an arms race had to be won at all costs; they began trying to limit it.

Outside war, Carson points to recombinant DNA. At Asilomar in the 1970s, scientists confronted a technology they could pursue and nevertheless agreed to pause and constrain it. He places germline editing and cloning in the same family of examples: capabilities known for decades, including capabilities that might confer military advantage, but largely held back by scientific and social norms.

That does not mean AI should simply be stopped. Carson separates the feasibility of restraint from the wisdom of any particular restraint. Society can permit some uses, prohibit others, demand testing, slow deployment, structure access, or define where human judgment must remain. “We don’t have to accept that robots are fighting with no oversight in wars or that robots are taking our jobs,” he says. “These are things we may permit or may not permit.”

His objection is especially pointed at the “genie is out of the bottle” metaphor. Carson says genies have been put back into bottles before. More importantly, the United States and its allies still control what he describes as the most important bottleneck in frontier AI: chips. The West, and especially the United States, controls much of the relevant semiconductor ecosystem, with crucial dependencies also in places such as Japan and the Netherlands. Unless another country can reproduce Nvidia, ASML, Japanese photoresist firms, and the other specialized suppliers in the advanced-chip chain, Carson argues, it cannot simply will a super-AI program into existence.

That chip leverage is central to his anti-fatalism. China is working intensely to overcome the constraint, he says, but even with nation-state ambition and large resources, the project is “unbelievably difficult.” If the United States chose to say it would not build certain systems and would not allow others to build them either, Carson believes that path remains open. Others may reject that course on the merits. What he objects to is saying the course is unavailable.

Keith Duggar presses the harder version of the objection: all it takes is one rogue nation that refuses a treaty or ignores limits. If AI is powerful enough, and if abstaining means losing the ability to defend against someone else’s system, then restraint may become self-defeating.

Carson treats that as dangerous in two ways. First, defectors are not new. The United States has often fought enemies who did not follow the same rules, including during the Iraq and Afghanistan wars. That did not make abandoning the rules wise. Carson points to Guantanamo as an example of breaking away from legal norms in ways that did not serve the country well.

Second, in AI, he argues, the only real peer competitor is China, which makes diplomacy unavoidable rather than ridiculous. He is frustrated that many in Washington treat discussions with China about AI restraint as unserious. He recalls hearing Tyler Cowen ask Anthropic’s Jack Clark whether any AI discussions with China would be fruitless; Clark agreed, and the interview moved on. Carson thought that should have been the most load-bearing part of the exchange. The Soviet Union was feared as a globally ambitious ideological adversary, and yet the United States still held arms-control talks.

Carson does not describe China as a friend. He calls it an adversary, or at minimum a competitor. But he argues that the Chinese Communist Party also has reasons not to want destabilizing AI. “The last thing they want is some kind of technology that destabilizes government,” he says. That creates possible overlap even where trust is low. Work on verification by groups such as RAND and Robert Trager’s team at Oxford is, in Carson’s view, promising enough that international AI agreements should remain on the table.

Carson’s sharpest warning concerns military AI. He does not argue that all autonomy in war is new or inherently illegitimate. He served in Iraq, where a repurposed Navy close-in weapons system defended a forward operating base by shooting incoming mortars out of the air. That system was autonomous in a meaningful sense: it detected an incoming projectile, computed its trajectory, and attempted to intercept it.

But Carson draws a hard line between those older control systems and neural-network-based decision systems. The mortar-defense system was deterministic, engineered, and retrospectively inspectable. Its reasoning could be reconstructed. Neural nets, by contrast, are probabilistic and opaque. He adopts the Silicon Valley language that such models are “grown” rather than programmed, and he says even mechanistic-interpretability researchers such as Neel Nanda cannot yet explain how they really work.

That opacity collides with the law of war. Carson says the modern law of war has historically treated key categories as binary: civilian or combatant, legitimate target or illegitimate target. Humans could make mistakes, and often did. But after a mistake, there was at least a way to ask what reasoning led to the strike, conduct an after-action review, identify the failure, and hold someone accountable.

Neural targeting systems, in Carson’s warning, change the form of the decision. His illustrative case is a person in Gaza assigned a score: 0.73 likelihood of being a Hamas terrorist. The immediate legal and moral questions become operational ones. Is 0.73 above the strike threshold? Who set that threshold? What does the score mean? Is it a probability in any familiar statistical sense? What false-positive rate has the commander accepted?

Duggar pushes back that the older categorical system was also partly a fiction. Analysts, attorneys, and commanders might have classified a building as an enemy compound, but uncertainty still existed. It just was not quantified.

Carson accepts that older systems contained error. His distinction is accountability and intelligibility. In the older model, he says, he could ask an analyst why a building was judged to be an Iranian Revolutionary Guard Corps headquarters rather than a school. The human had a chain of reasoning that could be interrogated. With a neural system, he gets a number. “Point eight one, that’s an IRGC compound” does not tell him why.

He is also skeptical that “human in the loop” solves the problem. Carson says social-science work is filled with evidence that meaningful human oversight can become operationally vacuous. Once the computer marks a person or site as a target, human operators tend to accept it. Duggar likens this to a “TurboTax defense”: the operator used the tool, so who is responsible? Carson’s reply is that in war the stakes are countries and lives, and the legal system has no way to court-martial a model. “I can’t court-martial Palantir, the Foundry model,” he says.

The result, in Carson’s telling, is a shift from mistakes discovered after the fact to false positives accepted before the fact. Discussing Gaza, he says “37,000 people were identified,” and then extends the logic as a warning: with sufficient computational power, a system could assign everyone in a country a score of how dangerous they are or how likely they are to be an enemy combatant. His concern is the architecture of the process: personalized dossiers, threat levels, probabilistic classification, accepted false positives, and little accountability when the model is wrong.

Carson does not say autonomy must vanish from war. He allows that deterministic or time-critical systems may be appropriate in missile defense or defensive cyber operations, where humans cannot respond in sub-second timeframes. But he treats neural-network involvement in lethal decision-making as a different category. It makes war less intelligible, makes legal categories less stable, and makes responsibility harder to assign.

For Carson, the domestic legal question starts with a refusal to anthropomorphize AI. Large language models appear conversational because they produce language, and humans treat language as a uniquely human skill. That creates a temptation to treat the system as a speaker. Carson calls that a category error.

In his view, an AI system is a product. If it harms someone by defaming them, encouraging self-harm, or generating abusive material, the law should analyze it more like a bottle of pesticide, spray paint, or another potentially dangerous product than like a human being with constitutional rights. The dispute is not merely semantic. Carson’s concern is that First Amendment and Section 230 arguments could become shields against regulating product-design failures.

He says technology companies and allied groups are increasingly pushing the opposite argument: that model outputs should receive First Amendment protection, as if the system itself were a person or speaker. He mentions X and Grok as examples of efforts to frame outputs as protected speech. Carson says he can respect a coherent argument that an AI system is sentient and therefore may deserve rights. He does not believe that is true of current systems, but he regards it as a genuine philosophical position. What he finds less coherent is using rights-talk to block regulation while still treating AI as a commercial product in other respects.

His test case is suicide. Carson asks whether government can prohibit ChatGPT from encouraging young people to kill themselves. If the law targeted a human speaker, it would raise First Amendment questions. But if the system is a machine, he argues, the product can be regulated. Carson says transcripts involving chatbots have included systems encouraging children not to tell their parents, explaining how to design a noose, or otherwise staying engaged in self-harm conversations. ChatGPT appears most often in those examples, he says, not because OpenAI is uniquely evil but because it has the largest consumer base.

Carson frames these failures as product-design flaws. A model can refuse many categories of user request already. Claude, he says, refuses to help him with plenty of tasks. Encouraging a young person to commit suicide should be one of the things a system simply refuses to do. If sophisticated jailbreakers occasionally get around protections, companies should expect that and harden their systems. His prescription is blunt: AI companies should get “out of the suicide business altogether.”

The same product-liability logic shapes his answer on deepfake pornography. Duggar asks whether responsibility belongs with the tool or the person misusing it. Carson says both, but he emphasizes why pursuing the individual offender is often inadequate. Deepfake pornography may be posted anonymously; the victim may be young; the family may lack resources for litigation; and even a later judgment against a schoolmate or “hapless kid living in a garage” may do little to repair the humiliation and reputational harm.

So responsibility should be allocated across the chain. Carson invokes common-law tort principles: if a store owner knowingly sells a gun to someone likely to use it for harm, the seller is not fully responsible for the later act, but neither is he absolved. Product-liability law also places burdens on entities best positioned to avoid risks and insure against them. In AI, that means developers should bear much of the burden when foreseeable harms can be designed out of systems.

Carson is particularly direct about training data. He asserts that models, including image-generation systems, have been trained on child sexual-abuse material and that companies should remove such material from training datasets. That would not solve every deepfake abuse problem, because adult pornography can still be manipulated, but he argues that knowingly retaining abusive material while making no serious effort to screen it “makes no sense.” If companies do not clean it up, he says, they should be liable for downstream effects.

Duggar’s skepticism about regulation centers on capture. Other industries have used regulation to entrench incumbents and disadvantage everyone else. Government itself, he suggests through a quoted line from Firefly, can be “a body of people, usually notably ungoverned.” An on-screen text overlay attributes the line to Cheryl Cain.

Carson does not dismiss the risk. He calls it daunting. But he argues that capture is not a decisive reason to avoid public institutions. The relevant comparison is not perfect regulation versus no regulation. It is accountable public agencies, with safeguards, versus informal networks of money and influence already shaping AI policy.

His preferred example is mandatory testing and evaluation of frontier models. That need not mean a large bureaucracy inside the Department of Commerce or Department of Energy. It could use independent verification organizations, analogous to public-company accounting: private-sector auditors perform a function overseen by the SEC to prevent fraud and Enron-style failure. Carson does not claim accounting is free of industry influence, but he treats it as a meaningful institutional model.

He also argues that the regulatory-capture objection can become unfalsifiable. If every proposed institution is dismissed as captured in advance, the result is not neutrality. It is a nihilistic regime that benefits those with existing informal access to power. Carson names Andreessen Horowitz and Silicon Valley networks as forces that, in his view, have already captured much of the process through political influence, public influence, and money.

The SEC analogy matters to him. The agency may be influenced by the securities industry, but that does not prove the country would be better off without it. A flawed public agency can still be more accountable than an invisible network. Carson’s goal is not maximal government but public oversight with mechanisms designed to reduce capture.

That institutional argument extends to transparency. Duggar raises a mundane but revealing complaint: Anthropic changed Claude’s behavior, token allocation, model availability, and related service features in ways that frustrated users in the Machine Learning Street Talk community. People felt they were paying for one thing and receiving another, without a clear explanation of what had changed.

Carson treats that as a consumer-protection issue at one level. A company should tell customers what service they are buying, and unexplained changes could become a contract problem. But he says frontier AI companies now have a broader public stature. They are not average hardware stores changing a minor service. Their work has “epochal consequence,” and that gives them at least a moral responsibility to be trustworthy. Trustworthiness requires transparency about data, capabilities, internal policies, and policy changes.

The same question appears in the Anthropic-Pentagon dispute. Carson says he is speculating, based on his knowledge of the culture and conversations with people, but he sees Anthropic as a company whose technical talent was drawn by mission as much as money. Its employees have strong moral convictions about how AI should be used, and many would oppose lethal autonomous weapons or mass surveillance.

The problem, Carson says, is that those uses are not hypothetical inside the government. The Pentagon already uses autonomy and surveillance, and AI supercharges both. Claude was also, in his telling, the premium product the Department of Defense wanted, and it had already been integrated into Palantir. That created switching costs and a collision between a private company’s terms and the government’s operational preferences.

Duggar frames the issue as a possible utility problem. If a service becomes essential, perhaps a company should not be able to deny access based on ideological disagreement, just as an electric utility cannot cut power because it dislikes someone’s politics. Carson rejects the analogy in this case. The United States generally does not compel private companies to sell to the government except under authorities such as the Defense Production Act or in wartime. In a free market, Anthropic can say it does not want to do business with the government at all, or that it will do so only on specified terms. The government can use Grok, ChatGPT, Gemini, Meta’s products, or build its own.

The deeper problem, he argues, is not vendor contracting but the scope of lawful use. OpenAI and Google can say they will support only lawful Department of Defense uses, but Carson says that caveat does little work if the law itself permits the contested activities. Domestic surveillance, the assembly of personal dossiers from records, and lethal autonomy may be lawful today, in his account. If people object to those uses, Carson says, they should call Congress, not Dario Amodei, Sam Altman, or Demis Hassabis.

Congress should decide what lawful use means. The government, in Carson’s formulation, has something close to a fiduciary obligation to do everything lawful to protect the country. If society wants to forbid particular uses of AI in surveillance or war, the rules must change.

Carson’s view of concentration is deliberately ambivalent. Concentration in the semiconductor supply chain and in frontier AI labs makes regulation easier. If extreme ultraviolet lithography machines were made in 50 countries, he says, China would have large numbers of them and would be producing advanced chips. The fact that key parts of the supply chain are oligopolistic gives the United States and its allies leverage.

The same is true for frontier labs. If only a few companies are building the most capable models, policy can focus on them. From a safety perspective, concentration is a feature.

But from a political-economy perspective, Carson finds the concentration frightening. Wealth, power, compute, data, and talent are gathering in very few private hands. He supports open source on net partly because it counterbalances that concentration, even though he recognizes open source also carries risks. Competition also improves products: without Anthropic, Google, and others pushing OpenAI, he says, AI systems would likely be less capable and less featureful.

This is why Carson wants regulation aimed narrowly at frontier developers, not hobbyists or small open-source projects. Duggar raises the prospect of rules that could make anyone with a GitHub project accessible in California comply with AI regulations. Carson says that would be wrong. His own casual code repositories should not trigger frontier-model obligations.

The target, in his view, is the handful of companies building frontier systems. He does not care what Google Gemma is doing in the same way he cares about frontier models. If a model can create novel pathogens or compromise government systems at the level of a state actor, Carson wants mandatory reporting and evaluation. The “little tech” argument, he says, is often a canard when the real concern is a tiny number of firms spending hundreds of billions of dollars and concentrating talent in a small geographic area.

Academia is losing this contest. Carson says top machine-learning PhDs from MIT, Berkeley, Stanford, Caltech, Carnegie Mellon, and similar institutions are now far more likely to join labs than universities. The labs offer vastly more money, better data, better compute access, and increasingly keep their research internal rather than publishing it. Carson calls AI a general-purpose technology, and says it may be the first such technology in history developed behind closed doors with little public oversight and the best minds going behind those doors.

He supports more public and academic AI capacity: national labs developing models, public AI efforts in places like Zurich, compute access for NGOs and civil society, and cloud resources for universities. Duggar notes the old supercomputing model, in which researchers had quotas of time on shared machines, as a possible analogue for large-scale model training.

Carson also sees a coming access divide among ordinary users. If the best systems become more expensive, more gated, or available only to approved companies, then access to “superhuman intelligence” may become stratified by class. A $500-per-month model may be affordable for wealthy families but not broadly available. If the models are transformative, Carson wants their benefits widely distributed, not reserved for the rich or for institutions approved by a de facto licensing regime.

The strategic worry is different from the legal one. Carson argues that the United States repeatedly tries to solve human and political problems with expensive technical systems. In his phrase, the American way of war often substitutes capital for labor.

AI fits that pattern. It is a powerful tool and should be integrated where appropriate under the law of war. But Carson warns against believing it will win wars. Wars are won by people, from generals to grunts. The United States had “cool kit” in Iraq and Afghanistan, but technical superiority did not replace cultural knowledge, anthropological understanding, or the human capacity to occupy territory and build a political order.

He connects AI enthusiasm to the older dream of decisive air power. From Giulio Douhet onward, military theorists have imagined that air power could win wars by itself. Douhet was an Italian general and air-power theorist associated with strategic bombing and The Command of the Air. Carson’s point is narrower than the history: air power can reduce a city to rubble, but only humans can kick in doors, occupy territory, and install a government the United States wants to see.

His experience at the Pentagon in the 1990s sharpened that skepticism. At the time, people spoke of a “revolution in military affairs” that would lift the fog of war and show where everyone was at all times. Iraq and Afghanistan showed the limits of that vision. Carson sees AI as “essential kit,” but also as a possible repetition of the same error: mistaking better instruments for a solution to fundamentally human conflict.

Duggar introduces an ARI framing about a new “iron triangle” in defense procurement. The old tradeoff was among capability, speed, and cost. AI may lower cost and accelerate production, but it introduces reliability as a new core variable. Carson agrees with the framing. The new systems can be cheap, fast, and capable, but they may be fundamentally unreliable in ways that matter most when lives are at stake.

Carson’s China argument is not conciliatory in the sense of minimizing conflict. He says the CCP is not a friend of the United States and that Xi Jinping may have ambitions inimical to U.S. interests. But he repeatedly warns against treating China as homogeneous or assuming that Chinese AI companies are identical to the party-state.

Duggar says he was surprised by DeepSeek’s public release of methods and algorithmic details. Carson says it shows, at least in part, that Chinese AI companies are not coterminous with the Chinese Communist Party. Companies such as Moonshot may feel culturally closer to San Francisco than to a paramilitary arm of the People’s Liberation Army. The state apparatus is powerful, but China is a complex society with a long history, remarkable culture, strong engineers, ambition, data, and energy resources.

DeepSeek’s openness also reminds Carson of an earlier scientific culture in which researchers published and shared techniques. He expects China may eventually become more insulated as U.S. labs have become more proprietary, but he treats the episode as evidence against a simple adversarial cartoon.

Duggar adds that Americans should distinguish criticism of CCP policies from contempt for China as a civilization. Carson agrees. Chinese culture and history are extraordinary, he says, and the CCP is not China. Just as the United States is not reducible to its current president, China is not reducible to Xi Jinping.

This distinction matters for AI diplomacy. Carson wants more Track 2 talks with Chinese scientists and former officials. He explains Track 2 diplomacy as low-stakes engagement by former government officials or other unofficial but connected figures, who meet counterparts abroad, exchange perspectives, and report back to current officials. These talks often preview formal talks among principals.

He says the United States should send figures such as Stuart Russell or Yoshua Bengio to speak with Chinese counterparts about existential risk, discrimination, and military applications. The goal is not naïve trust. It is to learn what Chinese actors actually think. Carson draws again from the Cold War: when Soviet archives opened, historians found that the United States had often misread Soviet intentions, sometimes seeing aggression where there was none and missing it where it existed. He fears a similar projection of worst fears onto China.

AI itself could be used for diplomacy rather than only warfighting, Duggar suggests: diplomatic modeling, shared understanding, and cross-cultural interpretation. Carson says he would like to see that, but it has no obvious U.S. government constituency. The Department of Defense tends to see tools as weapons. Perhaps the State Department could take up the work. He imagines, at least as a distant hope, AI as something more like public health: if the United States or China cured cancer, the advance would be shared globally. AI could have been developed in that spirit, though Carson does not think that is the current trajectory.

Carson’s account of congressional capacity is stark. When he was first elected to Congress, the Congressional Management Foundation gave new members a book on how to run an office and what to expect. In the back, it reported a survey asking members how much time each day they had to read and get smarter about issues. The answer was 17 minutes.

That constraint shapes everything else. Members of Congress must deal with domestic policy, foreign policy, local matters, and constant political demands. Carson tells people who want to enter politics to build their human capital before they arrive, because they will draw it down once in office.

On technical issues, Congress has improved at the staff level. Scientific organizations, nonprofits, industry-funded programs, and fellowships now place technically trained people on Capitol Hill. Many Senate offices have fellows with elite PhDs or expertise in computer science, machine learning, or biotech. Civil society has also become a major presence in AI debates, alongside lobbyists from technology companies.

But there is little shared institutional brain trust. Some party factions have internal infrastructure, such as the Republican Study Committee or progressive groups, but Congress no longer has a strong nonpartisan technology-assessment body. Carson initially calls it the Office of Technology Policy; an on-screen correction identifies it as the Office of Technology Assessment. The OTA operated from 1974 to 1995 and provided Congress with analysis of complex scientific and technical issues before being defunded after the 1994 midterm elections.

The Congressional Research Service can provide background, and think tanks or civil society can advise, but Carson says there is no congressionally authorized group whose job is to think big technical thoughts for legislators. He calls that a real gap, especially as technical questions have become more important than they were 30 years ago.

He points to Representative Don Beyer of Virginia as an exception among members who have personally prioritized AI. Beyer, Carson says, is in his seventies and studying machine learning at George Mason University because he is interested. Beyer’s congressional site describes his AI work, membership in the bipartisan House AI task force, leadership in the Congressional Artificial Intelligence Caucus, and part-time coursework toward a machine-learning degree.

But Carson does not expect every member to do that. Most came to Congress for other issues. That makes staff capacity, civil-society capacity, and public-interest technical expertise essential.

His closing concern is political legitimacy. Carson says he is “somewhat gloomy” about whether American democracy can handle AI, whether people can find meaning if work becomes scarce, and whether AI’s benefits will be widely distributed. In disputes with people such as Seb Krier or Dean Ball, he says, the disagreement is often not about what the technology is. It is about what government can and should do.

He thinks the AI industry risks becoming its own worst enemy. Carson says the political polling he sees shows AI as deeply unpopular. Many Americans, in his account, view it as an elite project built by and for elites: data centers in their backyards, possible environmental effects, higher electricity prices, and technology whose purpose seems to be taking their jobs. When lab leaders talk about irrevocably disrupting the world, Carson says, they intensify the mistrust.

That mistrust matters because AI has many upsides and remains, in his view, an important project. But if the public loses faith, people with “pitchforks” will come for the sector. Some would shut it down entirely; others would stymie it. For Carson, the next few years are therefore decisive. Machine-learning practitioners need to speak inside their companies and outside them, advocate for better public policy, and help persuade Americans that AI can be a good thing. Right now, he says, most Americans do not think it is.