AI's Red Lines Tested in Anthropic's Showdown With the Pentagon

Last month, US Defense Secretary Pete Hegseth delivered an ultimatum to Anthropic Chief Executive Dario Amodei: Allow the unrestricted use of the company's AI for "all legal purposes" within defense or see the company officially blocked from use by the US government and labeled a "supply chain risk."

Amodei publicly rejected the ultimatum, setting up the first confrontation of its kind between the government and a major company that is a highly respected AI developer. Rival OpenAI further escalated focus on the controversy by beating a path to the Pentagon and signing a deal to replace Anthropic in providing AI technology.

Amodei's refusal centered on two specific uses for Anthropic's Claude AI models that he said he is unwilling to permit: mass domestic surveillance of Americans and fully autonomous weapons without human control. His rationale was based on what is considered "ethical use" of advanced technology.

In 2016, before the modern large language model was even invented, the Future of Life Institute sponsored a now-landmark petition against the development of fully autonomous weapons. It was signed by 34,000 individuals, including Stephen Hawking, Elon Musk, Demis Hassabis (chief executive of Google DeepMind) and the so-called three "godfathers of AI" – Yann LeCun, Geoffrey Hinton and Yoshua Bengio.

The petition's warning is worth revisiting.

"Autonomous weapons will become the Kalashnikovs of tomorrow," it said. "They will become ubiquitous and cheap for all significant military powers to mass produce. It will only be a matter of time until they appear on the black market and in the hands of terrorists, dictators and warlords wishing to perpetrate ethnic cleansing. We therefore believe that a military AI arms race would not be beneficial for humanity."

Within hours of the Anthropic ban, OpenAI Chief Executive Sam Altman announced a new deal with the Department of Defense to deploy their models on classified networks. Altman later apologized that the timing looked "opportunistic," though he didn't rescind the deal. Caitlin Kalinowski, who oversaw hardware at OpenAI, subsequently quit, saying the company didn't take enough time before agreeing to let the Pentagon deploy its AI models on classified cloud networks.

OpenAI said it upholds the same "red lines" as Anthropic and its contract stipulates use of its AI system for all lawful purposes. OpenAI's wording regarding autonomous weapons is: "The AI system will not be used to independently direct autonomous weapons in any case where law, regulation or department policy requires human control."

However, law, regulation and department policies can be changed.

In a subsequent internal memo obtained by The Information, Amodei accused Altman of being "mendacious." He argued that since the ban on autonomous weapons exists only as a defense department policy that can be easily overturned, OpenAI has, in effect, placed no real limitations on the military. Amodei accused Altman of pretending to uphold ethical boundaries while doing nothing substantive to enforce them, and said his company will challenge the designation "supply chain risk" in court.

Amodei's tone is driven by the core convictions that led him to found Anthropic in 2020. At the time, Amodei was OpenAI's director of research and a pioneer in the ChatGPT series. OpenAI was a different organization back then. Its 2015 founding announcement stated the goal of seeking to become "a leading research institution that can prioritize a good outcome for all over its own self-interest."

In 2014, philosopher Nick Bostrom postulated in "Superintelligence: Paths, Dangers, Strategies" that superintelligent AI poses an existential risk to humanity. Both Altman and Amodei felt the systems they were creating had the potential to become this very intelligence. When OpenAI transitioned to a "capped-profit" entity in 2019, after a billion-dollar investment from Microsoft, the focus pivoted from ensuring a "good outcome" to making money. Feeling OpenAI had betrayed its mission, Amodei left to start Anthropic, a culture still steeped in ethics.

The systems we see in 2026 – AI capable of making decisions and taking actions without human input – are a quantum leap beyond simple chatbots. We are approaching Bostrom's definition of superintelligence: an agent able to act in the world more effectively than any human. If a creator believes its AI possesses that kind of power, why would it grant the autonomous authority to use lethal force? Is humanity comfortable giving a machine the power of life and death on the battlefield? This, however, is exactly where military AI is heading.

How will this situation evolve? History provides a grim template. In 2017, the Pentagon and Google launched Project Maven, using machine learning for object recognition in surveillance data. Despite large-scale employee protests and walk-outs that forced Google to withdraw in 2018, the project did not die. Palantir, the intelligence firm founded by Peter Thiel, took over. Since then, Maven has been deployed in the Russia-Ukraine conflict, airstrikes in Iraq and Syria, the capture of Nicolás Maduro in Venezuela and in current tactics in Iran.

We can imagine, sadly, that Anthropic's refusal will not stop the proliferation of autonomous weapons. Ultimately, the question returns to a fundamental issue: Is AI simply a new tool for information and profit, or is it a powerful new form capable of immense harm?

If human control over AI-powered weapon systems is eliminated, we are simply playing with fire that threatens humanity.

(The author has multiple years of experience working at leading AI companies in the US, focusing on data preparation and large-scale distributed model training.)