Experts warn of AI-powered autonomous weapon risks

For decades, the military has used autonomous weapons such as mines, torpedoes, and heat-guided missiles that operate based on simple reactive feedback without human control. However, artificial intelligence (AI) has now entered the arena of weapons design.

According to Kanaka Rajan, associate professor of neurobiology in the Blavatnik Institute at Harvard Medical School, and her team, AI-powered autonomous weapons represent a new era in warfare and pose a concrete threat to scientific progress and basic research.

AI-powered weapons, which often involve drones or robots, are actively being developed and deployed, Rajan said. She expects that they will only become more capable, sophisticated, and widely used over time due to how easily such technology proliferates.

As that happens, she worries about how AI-powered weapons may lead to geopolitical instability and how their development could affect nonmilitary AI research in academia and industry.

Rajan, along with HMS research fellows in neurobiology Riley Simmons-Edler and Ryan Badman and MIT Ph.D. student Shayne Longpre, outline their central concerns—and a path forward—in a position paper published and presented at the 2024 International Conference on Machine Learning.

In a conversation with Harvard Medicine News, Rajan explained why she and her team decided to investigate the topic of AI-powered military technology, what they see as the biggest risks, and what they think should happen next.

You are a computational neuroscientist who studies AI in the context of human and animal brains. How did you end up thinking about AI-powered autonomous weapons?

We started considering this topic in reaction to a number of apocalyptic predictions about artificial general intelligence circulating in spring 2023. We asked ourselves, if those predictions are indeed blown out of proportion, then what are the real risks to human society? We looked into how the military is using AI and saw that military research and development is pushing heavily toward building systems of AI-powered autonomous weapons with global implications.

We realized that the academic AI research community would not be insulated from the consequences of widespread development of these weapons. Militaries often lack sufficient expertise to develop and deploy AI tech without outside advice, so they must draw on the knowledge of academic and industry AI experts. This raises important ethical and practical questions for researchers and administrators at academic institutions, similar to those around any large corporation funding academic research.

What do you see as the biggest risks as AI and machine learning are incorporated into weapons?

There are a number of risks involved in the development of AI-powered weapons, but the three biggest we see are: first, how these weapons may make it easier for countries to get involved in conflicts; second, how nonmilitary scientific AI research may be censored or co-opted to support the development of these weapons; and third, how militaries may use AI-powered autonomous technology to reduce or deflect human responsibility in decision-making.

On point one, a big deterrent that keeps nations from starting wars is soldiers dying—a human cost to their citizens that can create domestic consequences for leaders. A lot of current development of AI-powered weapons aims to remove human soldiers from harm’s way, which by itself is a humane thing to do. However, if few soldiers die in offensive warfare, it weakens the association between acts of war and human cost, and it becomes politically easier to start wars, which, in turn, may lead to more death and destruction overall. Thus, major geopolitical problems could quickly emerge as AI-powered arms races amp up and such technology proliferates further.

On the second point, we can look to the history of academic fields like nuclear physics and rocketry. As these fields gained critical defense importance during the Cold War, researchers experienced travel restrictions, publication censorship, and the need for security clearance to do basic work. As AI-powered autonomous technology becomes central to national defense planning worldwide, we could see similar restrictions placed on nonmilitary AI research, which would greatly impede basic AI research, worthwhile civilian applications in health care and scientific research, and international collaboration. We consider this an urgent concern given the speed at which AI research is growing and research and development on AI-powered weapons is gaining traction.

Finally, if AI-powered weapons become core to national defense, we may see major attempts to co-opt AI researchers’ efforts in academia and industry to work on these weapons or to develop more “dual-use” projects. If more and more AI knowledge starts to be locked behind security clearances, it will intellectually stunt our field. Some computer scientists are already calling for such drastic restrictions, but their argument dismisses the fact that new weapons technologies always tend to proliferate once pioneered.

Why do you think weapons design has been relatively overlooked by those thinking about threats posed by AI?

One reason is that it’s a new and quickly changing landscape: Since 2023, a number of major powers have begun to rapidly and publicly embrace AI-powered weapons. Also, individual AI-powered weapons systems can seem less threatening in isolation, making it easy to overlook issues, than when considered as a broader collection of systems and capabilities.

Another challenge is that tech companies are opaque about the degree of autonomy and human oversight in their weapons systems. For some, human oversight could mean pressing a “go kill” button after an AI weapons unit makes a long chain of black box decisions, without the human understanding or being able to spot errors in the system’s logic. For others, it could mean a human has more hands-on control and is checking the machine’s decision-making process.

Unfortunately, as these systems get more complex and powerful, and reaction times in war must be faster, the black box outcome is more likely to become the norm. Furthermore, seeing “human-in-the-loop” on AI-powered autonomous weapons may lull researchers into thinking the system is ethical by military standards, when in fact it does not meaningfully involve humans in making decisions.

What are the most urgent research questions that must be answered?

While a lot of work is still needed to build AI-powered weapons, most of the core algorithms have already been proposed or are a focus of major academic and industry research motivated by nonmilitary applications—for example, self-driving vehicles. With that in mind, we must consider our responsibility as scientists and researchers in ethically guiding the application of these technologies and how to navigate the effects of military interest on our research.

If militaries around the world aim to replace a substantial portion of battlefield and support roles with AI-powered units, they will need the support of academic and industry experts. This raises questions about what role universities should play in the military AI revolution, what boundaries should not be crossed, and what centralized oversight and watchdog bodies should be set up to monitor AI use in weapons.

In terms of protecting nonmilitary research, we may need to think about which AI developments can be classified as closed-source versus open-source, how to set up use agreements, and how international collaborations will be affected by the increasing militarization of computer science.

How can we move forward in a way that enables creative AI research while safeguarding against its use for weapons?

Academics have had and will continue to have important and productive collaborations with the government and major companies involved in technology, medicine, and information, as well as with the military. However, historically academics have also had embarrassing, harmful collaborations with the sugar, fossil fuel, and tobacco industries. Modern universities have institutional training, oversight, and transparency requirements to help researchers understand the ethical risks and biases of industry funding and to avoid producing ethically dubious science.

To our knowledge, no such training and oversight currently exists for military funding. The problems we raise are complex and can’t be solved by a single policy, but we think a good first step is for universities to create discussion seminars, internal regulations, and oversight processes for military-, defense-, and national security agency-funded projects that are similar to those already in place for industry-funded projects.

What do you think is a realistic outcome?

Some in the community have called for a full ban on military AI. While we agree that this would be morally ideal, we recognize that it’s not realistic—AI is too useful for military purposes to get the international consensus needed to establish or enforce such a ban.

Instead, we think countries should focus their efforts on developing AI-powered weapons that augment–rather than replace–human soldiers. By prioritizing human oversight of these weapons, we can hopefully prevent the worst risks.

We also want to emphasize that AI weapons are not a monolith, and they need to be examined by capability. It’s important for us to ban and regulate the most egregious classes of AI weapons as soon as possible and for our communities and institutions to establish boundaries that should not be crossed.