by Joseph A. Getto*
In the autumn of 1854, on the battlefields of the Crimean War, British forces under Field Marshal Lord Raglan faced Russian defenders near Sevastopol. When enemy troops seized several artillery positions, Raglan — watching from a distant ridge — dispatched a vague order demanding that cavalry “advance rapidly to the front” and prevent the enemy from carrying away a specific set of guns. The order made perfect sense from his elevated vantage point. It made no sense at all to the officers in the valley below, who could see only one set of guns: a heavily fortified Russian battery at the end of a mile-long killing field.
It fell to Lord Cardigan to lead the 670 men of the Light Brigade in a suicidal charge. Within twenty minutes, 278 of the men were dead or wounded, with a casualty rate near 40%. It is considered one of the most notorious blunders in military history. The disaster had a simple cause: the impossibility of fully understanding an order issued by someone who did not share the context or perspective of those executing it.
The lesson of that battle appears entirely lost on the present-day Pentagon.
In 2025, the Department of Defense awarded contracts to multiple large language model developers to embed AI into everyday use in the military with a goal to provide chatbot administrative assistants to officers, many of whom were already using such tools informally. Since then, the Pentagon’s ambitions for AI have widened. With the technology shifting from chatbots into command-and-control decisions, the Trump administration has pushed the AI company Anthropic to remove what the industry calls “Human-in-the-Loop” (HITL) requirements. These safeguards prevented Anthropic’s AI program Claude from being used to power fully autonomous weapons. Anthropic refused, and the Pentagon’s response was to designate Anthropic a “supply chain risk” and order Claude removed from all military systems. In the military’s framing, HITL was deemed an “unacceptable” operational delay.
Defenders of the current trajectory argue that AI is merely providing “decision support” — that a human remains responsible for any decision. This framing is dangerously misleading. Large Language Models (LLMs) tend to push conflicts toward escalation and are extraordinarily prone to overconfidence errors. Studies claiming that certain AI models are statistically more likely to initiate nuclear launches should be treated with both skepticism and concern, but the subtler danger is just as serious: an AI system, even in recommending what it calculates to be a less escalatory option, may still set catastrophe in motion. These models cannot explain their own reasoning to their developers, let alone to a battlefield commander who needs to understand why the system just recommended a strike on a particular set of coordinates. Compressed timetables, opaque logic and machine confidence are a volatile mixture. What begins as a conventional escalation can rapidly snowball into a nuclear crisis through an accumulation of smaller escalations and miscalculation.
A deeper problem underlying all of this is automation bias: the well-documented tendency for human operators to defer their decisions to machine recommendations. Students of the Cold War know well that this bias nearly ended the world. In 1983, the Soviet Oko early-warning satellite system malfunctioned and reported that the United States had launched a nuclear strike. The officer monitoring the system, Stanislav Petrov, was rightly skeptical of a system he knew to be new and untested. He discounted the alert and did not report it up the chain of command. We are fortunate he had only one alarm to weigh and the time to doubt it. Had Petrov been interfacing with a modern AI decision-support system, he might have faced dozens of simultaneous high-confidence alerts, each demanding acknowledgment. What might he have done then?
These consequences of automation are not theoretical. AI target identification systems have demonstrated a disturbing tendency to misclassify civilians as military targets, optimizing for the pattern they were trained to find and finding it everywhere. When a human operator is asked to approve a thousand targeting recommendations in a single day, they are not deliberating — they are rubber-stamping. This pattern has already been documented in Israel’s war in Gaza, where the volume and speed of AI-assisted targeting raised serious questions among human rights investigators about whether meaningful human review was possible at all. A more recent example may be the U.S. strike on a school in Iran just two months ago, which left over 170 dead, almost all of them children and educators.
A misidentification, or an AI-suggested maneuver that routes a combat unit into a kill zone is an ever-increasing inevitability. When it comes to pass, such an order will be the modern equivalent of Raglan’s order reaching Cardigan: a veritable Charge of the Silicon Brigade.
Abandoning AI in defense applications does not seem to be a realistic solution. The Pentagon has invested too heavily in the technology both financially and strategically for a reversal to be viable. AI has also demonstrated real combat effectiveness on the more defined front lines of the later stages of the war in Ukraine, where it has helped supplement where it has helped compensate for exhausted forces in deterring Russian incursions. But its use must be bounded by the law, not by unilateral decisions made by the Pentagon or by Silicon Valley. Arms control experts, lawmakers and international partners must push urgently for enforceable restrictions.
Members of Congress have already drafted legislation to address these dangers. Most notably, Rep. Elissa Slotkin proposed a framework to impose meaningful oversight on the deployment of AI systems in military operations. That effort deserves broader support, and Congress should build on it. Whatever form oversight legislation ultimately takes, it must codify a minimum of five safeguards before implementation of these systems goes any further.
First, Human-in-the-Loop requirements must be preserved for any decision involving lethal force or escalatory action. Second, AI systems must provide human-readable explanations of their recommendations before those recommendations can be acted upon — a commander who cannot understand why a strike was recommended, or why a unit was directed to a particular location, cannot meaningfully authorize it. Third, every use of AI in a military operation must be documented and auditable. Just as conventional command decisions that lead to civilian deaths are legally required to face public accountability and scrutiny, so too must AI-assisted decisions that result in civilian casualties be published and audited. This transparency also serves a direct security purpose, allowing investigators to isolate data poisoning attacks and identify where a system’s reasoning failed before that failure repeats at scale. Fourth, AI models with access to classified operations must open their internal decision-making processes to those with appropriate clearance. Black-box systems –– a term that describes the inscrutable opacity in which these models think –– should have no place in military decisions. Finally, and most urgently, AI must be kept entirely out of the command-and-control architecture of the nuclear arsenal. As the Petrov incident illustrates, automation bias has already brought us to the edge of accidental nuclear war once. We must not engineer a faster path down that dangerous slope.
However, domestic safeguards can only do so much. The United States must work with allies and adversaries alike to establish binding international norms governing the use of AI in combat. The Convention on Certain Conventional Weapons and the Geneva Conventions offer a good foundation, but they were not written with autonomous weapons in mind. Luckily, a promising starting point already exists: the 2024 bilateral policy agreement between President Biden and President Xi, which included explicit HITL commitments for nuclear decision-making. Though rolled back by the Trump administration, this framework proved that such international policy alignments are achievable. Expanding it into a broader multilateral accord can be within reach if the political will exists to pursue it. American safeguards, applied unilaterally, will mean little if adversaries and allies alike continue to race to the bottom without them.
We are, right now, deciding how wars will be fought in the next century. If we do not take care in how we adopt and adapt to these technologies, we may find ourselves riding into our own valley of death.
This time, we may not have the luxury of a poem to commemorate us.
“Forward, the Light Brigade!”
Was there a man dismayed?
Not though the soldier knew
Someone had blundered.
Theirs not to make reply,
Theirs not to reason why,
Theirs but to do and die.
Into the valley of Death
Rode the six hundred.
— Alfred Lord Tennyson, “The Charge of the Light Brigade”
*Editor’s note: Writing for the Center’s new Next Up in Arms Control series, Joseph A. Getto is a recent Masters graduate in International Conflict Resolution at the Heller School for Social Policy and Management at Brandeis University in Waltham, Massachusetts. In 2025, he served as a Specialist Fellow in AI Safety, Nuclear Risk and Global Governance with the Alva Myrdal Centre for Nuclear Disarmament.
DISCLAIMER: Next Up in Arms Control is a way for the Center to present an opportunity for dialogue and provocation through the thoughtful exchange of ideas and opinions on new or different ways to reduce and eventually eliminate nuclear threats. Opinions are the authors’ alone and not necessarily reflective of Center’s positions on the issues addressed.
