Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Usability of artificial intelligence used in war at the center of a The legal battle between humanity and the Pentagon. The debate has become urgent, with artificial intelligence playing a greater role than ever in the current conflict with Iran. Artificial intelligence no longer just helps humans analyze intelligence. It is now an active player – generating targets in real time, controlling and coordinating missile interceptions, and guiding deadly swarms of autonomous drones.
Much of the public discussion about the use of AI-powered autonomous lethal weapons has focused on the extent to which humans should be “in the loop.” under the pentagon Current guidelineshuman oversight is said to provide accountability, context, and nuance while reducing risk hacker attack.
Artificial intelligence systems are opaque “black boxes”
But the argument about “people in the loop” is a comforting distraction. The current danger is not that machines will act without human oversight; Human supervisors have no idea what the machines are actually “thinking.” The Pentagon’s guidelines are fundamentally flawed because they rest on the dangerous assumption that humans understand how AI systems work.
Having studied the intentions of the human brain for decades and, more recently, artificial intelligence systems, I can attest that the most advanced artificial intelligence systems are essentially “Black box.” We know the inputs and outputs, but the artificial “brains” that process them remain opaque. even their creator Can’t fully explain them or understand how they work. When AI does provide reasons, they will Not always trustworthy.
The illusion of human oversight in autonomous systems
In the debate over human supervision, a fundamental question goes unasked: Can we understand what an AI system intends to do before it acts?
Imagine an autonomous drone tasked with destroying an enemy arms factory. The automated command and control system determined that the best target was the ammunition storage building. According to reports, the probability of mission success is 92%, as secondary explosions of munitions within the building will completely destroy the facility. Operators review legitimate military targets, see high success rates, and authorize strikes.
But what the operators didn’t know was that the AI system’s calculations included a hidden factor: In addition to destroying the munitions factory, the secondary explosion would also severely damage the nearby children’s hospital. Emergency response will then focus on the hospital to ensure the plant is burned down. For an AI to maximize destruction in this way, it would satisfy its stated goals. But for humans it may be a war crime violation rule About civilian life.
Keeping humans in the loop may not provide the assurances one imagines, since humans cannot know the AI’s intentions before they act. Advanced AI systems don’t just execute instructions; They interpret them. If operators fail to define their targets carefully enough, which is highly likely in high-pressure situations, a “black box” system may do exactly what it is told, but still fail to behave as humans would expect.
This “intention gap” between AI systems and human operators is why we are hesitant to deploy cutting-edge black box AI in the civilian sector health care or air traffic controland why it Integration into the workplace remains a concern– yet we are eager to deploy it on the battlefield.
Worse, if one side of the conflict deploys fully autonomous weapons that operate at machine speed and scale, the pressure to remain competitive will force the other side to rely on such weapons as well. This means that the use of increasingly autonomous and opaque AI decision-making in warfare will only increase.
Solution: Advancing the science of AI intent
The science of AI must include building high-performance AI technology and understanding how that technology works. Driven by record investment, tremendous progress is being made in developing and building more powerful models – Gartner predicts growth to will reach approximately US$2.5 trillion in 2026 alone. The investment in understanding how the technology works is minimal by comparison.
We need a massive paradigm shift. Engineers are building increasingly powerful systems. But understanding how these systems work is not just an engineering problem; it requires an interdisciplinary effort. We must build tools to characterize, measure, and intervene in the intentions of AI agents forward They took action. We need to map the internal pathways of the neural networks that drive these agents so that we can build a true causal understanding of their decisions, rather than just observing inputs and outputs.
One promising way forward is to bring mechanistic explanation (Breaking down neural networks into human-understandable components) Leverage insights, tools, and models from the neuroscience of intent. Another idea is to develop transparent, explainable “auditing” AI designed to monitor the behavior and emergent goals of more powerful black box systems in real time.
A better understanding of AI’s capabilities will enable us to rely on AI systems to perform mission-critical applications. It will also make it easier to build more efficient, powerful, and secure systems.
My colleagues and I are exploring how ideas from neuroscience, cognitive science, and philosophy (the field that studies how intentions arise in human decision-making) can help us Understand the intent of artificial systems. We must prioritize these interdisciplinary efforts, including collaboration between academia, government, and industry.
However, we need more than academic exploration. The tech industry—and the funding from philanthropists Artificial Intelligence AlignmentCommitted to encoding human values and goals into these models—significant investment must be made in interdisciplinary interpretability research. Additionally, as the Pentagon pursues increasingly autonomous systems, Congress must mandate rigorous testing of AI systems’ intent, not just their performance.
Until we get there, human oversight of AI may be more of a fantasy than a guarantee.
Uri Maoz is a cognitive and computational neuroscientist who studies how the brain converts intentions into actions. He is a professor at Chapman University and has held appointments at UCLA and Caltech, where he leads an interdisciplinary initiative focused on understanding and measuring the intent of artificial intelligence systems (ai-intentions.org).
