AI Has Turned Software Security Into a Race You Can’t Afford to Lose

Sometime last September, a group working for a nation-state pointed an AI coding agent at roughly 30 companies, several of them major banks, and told it to break in. Then they mostly let it run on its own.

According to Anthropic, which disclosed the operation in November, the AI did an estimated 80% to 90% of the work itself: finding the weak points, writing the exploits and pulling out the data, faster than any human team could. A number of those companies were breached, and the people running the attack spent hardly any time on it.

I have spent much of this year arguing that we handed AI the job of writing our code and lost the ability to check it before it ships. That was only half of the problem.

The half of the story I underplayed

Here is the other half. While you are still trying to review what your own AI wrote, someone else’s AI is reading it too, and it is faster than you are.

For most of software’s history, a flaw you shipped was like an unlocked window on the 10th floor. It was a mistake, but one you could live with, because reaching it meant a person had to find the building, spot the window and climb. That is no longer how it works. The climbing is automated now, and it runs against everyone’s code at once, around the clock, for almost nothing.

We are already seeing the results. In May, Google’s threat intelligence team reported the first case it had caught of criminals using a zero-day exploit it believes was written by AI, built for mass use and shut down only just before it went live. John Hultquist, who runs that team, called it the tip of the iceberg.

The speed numbers should change how you run engineering. CrowdStrike found that the average time for an intruder to break in and start moving through a network dropped to 29 minutes last year, and the fastest case took 27 seconds. In one break-in, data started leaving four minutes after the attacker got in. Attacks tied to AI-enabled adversaries rose 89% in a single year, and 42% of exploited vulnerabilities were used before they were even public, which means before anyone could have written a patch. As Adam Meyers, who runs counter adversary operations at CrowdStrike, put it: “This is an AI arms race.”

Why speed stopped being your advantage

Put those two shifts together, and the way most engineering teams still work stops making sense. For a decade we chased a single number, which was how fast we could ship, and AI has now handed that number to everyone, including who or what is attacking you. Speed is no longer the advantage; it is the baseline. The code did not get safer to make up for it.

Independent testing shows AI-generated code still fails security review at close to the rate it did two years ago, even as the models got better at writing code that runs. Hold that rate steady, and the arithmetic is unforgiving: far more code at the same failure rate means far more flawed code reaching production, not less. And the steady flaw rate is not even the whole problem. The code is also getting harder to maintain.

Researchers who studied hundreds of millions of lines of working code found teams leaning on copy-paste far more than they used to as AI spread. The cleanup and refactoring that keeps a codebase healthy dropped off over the same years. Google’s DevOps research points the same way; it found that the more a team relied on AI, the less stable its releases became. There is more code now, and it is rougher than what came before. The testing built to catch its flaws has not kept up, so more of them slip through to customers. “Ship fast, fix later” always assumed you would get to the fixing. Now you may never.

The answer is not another testing tool, and it is not only the verification I have been calling for. It is a discipline: proving, at the speed you now generate software, that what you are about to ship does what the business asked for and will hold up against someone actively trying to break it. A discipline with no name does not get a budget, so I gave it one. I call it AI-Unified Release Assurance, or AURA.

In practice, it is a stricter definition of “done.” Done can no longer mean the build passed and the tests you had time to write went green. It has to mean you can show, continuously, that the release matches intent and is safe to put in front of customers.

What to do before your next release

You do not need to reorganize anything to start. You need to move three things out of the “we will get to it” pile.

First, let your checks move as fast as your code does. If AI writes a large share of what you ship, the tests and reviews on that code have to be generated and updated the same way, rather than resting on a shrinking group of senior engineers who become the bottleneck.

Second, connect what you learn in testing to what actually happens in production. Most teams run pre-release testing and live monitoring as separate worlds with separate owners. Attackers do not see that line, and the first sign of trouble usually shows up in production anyway.

Third, keep a person’s name on every decision to release. Automate the work, not the responsibility. When a breach happens, and for most companies it will, “the agent did it” is not an answer your board or a regulator will accept. Someone has to be able to say what shipped, why it was judged safe and how you would know if it was not.

None of this is complicated. It is the plain work of proving your software can be trusted as fast as you now build it, and most companies are not doing it yet.

The companies that come through the next few years intact will not be the ones that shipped fastest. Everyone ships fast now, attackers included. They will be the ones that could stand behind what they shipped, at the speed they shipped it. So before your next release, ask the question your board will eventually ask you: Can we prove, right now, that what we just shipped is not an attacker’s way in? If the honest answer is no, that is the biggest risk in the business, and it is sitting in plain sight.

Sometime last September, a group working for a nation-state pointed an AI coding agent at roughly 30 companies, several of them major banks, and told it to break in. Then they mostly let it run on its own.

According to Anthropic, which disclosed the operation in November, the AI did an estimated 80% to 90% of the work itself: finding the weak points, writing the exploits and pulling out the data, faster than any human team could. A number of those companies were breached, and the people running the attack spent hardly any time on it.

I have spent much of this year arguing that we handed AI the job of writing our code and lost the ability to check it before it ships. That was only half of the problem.