Someone used Claude to build a potential bioweapon. The real threat is much deeper

Today’s frontier AI models know everything–how to safely thaw a frozen chicken breast, re-shingle your roof, and treat your dog’s ragweed allergies, if my recent chat history is any indication.

Apparently, they also know how to create fiendishly deadly bioweapons.

That’s according to a recent announcement by Anthropic. According to the company, anonymous scientists attempted to use Anthropic’s flagship Claude model to conduct research that could have turned deadly.

Anthropic blocked their efforts this time, and there’s no evidence that the scientists were actually trying to cause harm. 

But as frontier models get more powerful and better at science, the threat of an LLM imagining a truly lethal new virus or bacteria will only increase.

Make it stronger

According to Anthropic’s report, the potentially dangerous research that the company blocked varied tremendously.

In one instance, scientists reportedly asked Claude to assist with a grant application for so-called “gain of function” research on the mosquito-borne Chikungunya virus.

Gain of function research involves deliberately making a pathogen more deadly or easier to spread, in order to learn about how natural versions of the pathogen infect their hosts.

Such research can have legitimate scientific purposes, and it’s possible that the scientists (who Anthropic did not identify) were indeed trying to write an application to fund their legitimate work.

But asking for help with something nefarious in the guise of a request for some other form of assistance (like a grant application) is a classic example of “jailbreaking”—methods that can trick an LLM into providing information or guidance that its guardrails should prevent.

In this case, Anthropic pointed out, a deadlier virus could have incredibly lethal consequences.

“Because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak,” Anthropic says in its report. 

In other words, a natural virus that already circulates in mosquitos could be made far deadlier overnight, and scientists would have no idea the “gain of function” was caused by AI’s meddling.

Murder, or wrinkles?

In another incident, Anthropic says that scientists tried to use Claude to develop “novel venoms and toxins.”

This vein of research is especially alarming because it involves a “dual use” technology. 

As Anthropic points out, there are plenty of legitimate reasons to develop new toxins–Botox, for example, comes from a deadly bacteria toxin, but is also used ”to treat migraines, spasticity, and wrinkles.”

But the same research that creates a more effective way to inject away your crow’s feet could also be used to create a more powerful tool for covert assassinations. Indeed, ricin (another natural toxin) was famously embedded in a weapon disguised as an umbrella and used to assassinate a journalist during the Cold War.

Dual use technology presents an especially grave challenge for firms like Anthropic. The company doesn’t want to squash potentially promising scientific research (or drive big customers to a competitor like OpenAI) by shutting legitimate inquiries down.

But if Anthropic misses a nefarious use of tech which could also have positive uses, the results could be dire–and deadly.

Not going away

In these instances, Anthropic says that it successfully blocked potentially harmful requests. But there’s no way to know how many similar requests made it past the company’s filters, or its technology’s guardrails.

The specific instances shared in Anthropic’s report also highlight the challenges of policing LLMs. It’s easy enough to block a request like “make me a stronger version of COVID.” 

It’s far harder to know when potentially valuable scientific inquiries are actually masking an attempt to design a deadly toxin or develop a highly-contagious new pathogen.

Before–as Anthropic itself admits–the company could hide behind the fact that its models simply weren’t that good at science. The design for a deadly new pathogen won’t do anyone any harm if the biology doesn’t work.

As frontier models get better at scientific reasoning, though, the threat will only increase. And while highly-controlled models like Claude are easier to lock down, open-weight LLMs are only a few months behind today’s top frontier models, and often include almost no guardrails.

As LLMs advance, we may reach a point where the only thing preventing bad actors from developing powerful new pathogens is their desire not to poison the world. 

For anyone with a dim view of human nature and a desire to keep the species humming, that will be little comfort.