New anti-deepfake rules ignore a key risk AI creates for banks

• Key insight: Bankers in the U.S. can learn a few things from watching the rollout of the EU’s new AI Act. Specifically, they should notice the gaping hole where rules about identity verification ought to be.
• What’s at stake: The banking industry has spent years building systems around the idea that someone’s voice, and/or face, is sufficient enough evidence to verify who they are. Artificial intelligence completely challenges that assumption.
• Forward look: Europe has made a real start on the deepfakes that want to be seen. The ones that don’t are still out there, working away in the space that this law leaves behind.

Three years ago, VICE published an article that went viral. In it, reporter Joseph Cox detailed how he broke into his own bank account using a free tool. It took a few attempts and required him to feed samples of his voice into the platform, but eventually it worked. He was in.

Processing Content

Technology used to create deepfakes has only become more sophisticated since then, which is likely the reason this type of crime has increased year on year and why The FBI now tracks AI as its own fraud category.

For banking this creates a real challenge. This is an industry that’s spent years building systems around the idea that someone’s voice, and/or face, is sufficient enough evidence to verify who they are. Artificial intelligence completely challenges that assumption.

I am currently writing this from my living room in Amsterdam where I’ve built a career detecting deepfakes for forensics teams. The new anti-deepfake law I’m about to discuss is European, but American banks would be mistaken if they assume it has nothing to do with them. 

After all, AI-fraud in banking is happening on both sides of the Atlantic. Regardless of their location, banks are all investing in the same kinds of technology to try to stop this fraud. Europe may be a little bit further along in regulating it, but the U.S. will soon follow. So, this argument is a preview of what’s to come. 

The Act also applies to non-EU providers whose systems are used in Europe, so plenty of American vendors are already in scope.

Last week new EU regulations came into effect, introduced under Article 50 of the AI Act. They were greeted with a lot of praise. After all, on paper, it’s an extremely sensible step. One that I believe will do a lot of good in curbing the harm caused by AI.

The problem is, the deepfakes Article 50 is designed to catch are the ones meant to be noticed. Fraud, however, runs on the opposite logic. It needs discretion. 

Article 50 states that all synthetic content needs to have a machine-readable tag. But how plausible is it, really, that someone trying to hack into a bank’s bypass will own up to this, let alone hold the very marker used to identify them? They just won’t. 

That’s not a flaw in Article 50, but it is a boundary. The actual problem is what sits on the other side of that boundary.

Within Annex III of the act is information about what applications are considered to be high-risk. Biometrics are on that list, but there’s a caveat. 

Biometric identification, working out who somebody is by matching a face or voice against a database of many, is featured on the list. Biometric verification, checking one person against their own record, is not.

Here’s why this distinction matters. 

Read more:

Earlier this year a man stood trial in The Netherlands after successfully opening 46 bank accounts. He did so by using deepfake technology to blend his own features with faces from passport photos he’d received while carrying out a rental scam. 

The check the bank used is precisely the category the Act doesn’t regulate, biometric verification. It’s also increasingly being used as a way to onboard remote customers. 

It can be easy to assume that not having biometric verification classified as high-risk is good news for banks. After all, less regulation typically means less red tape. In this instance though, I don’t believe that to be the case.

Reason being that the high-risk requirements would eventually mean that providers that fall under them would need to demonstrate greater resilience against manipulation. Verification vendors won’t be required to do the same.

To complicate matters even more, those high-risk requirements were originally supposed to land at the same time as Article 50. But the Digital Omnibus delayed them until 2027, putting banks in a really odd position. 

For the next year, the rules telling them to label AI-generated materials are already in force. The ones that would hold the systems they use to account are not.

The same goes for the US. Fincen has warned banks about deepfake-enabled identity fraud. It even published red flags and told banks when they should report suspicious activity. Useful, as far as it goes. But it doesn’t ask anything of the companies building verification systems.

Let me be clear. I don’t think that any of the challenges mentioned above mean that Article 50 will fail. On the contrary, I really do believe that it will work wonders against synthetic content abuse.

I know what harm deepfakes can do. I founded my company after a close friend of mine had her own likeness stolen and used in a romance scam. So, any regulation that can help is absolutely welcome.

What I don’t believe though, is that the regulations are ready for the banking industry. We’re yet to find a good enough solution to this kind of fraud both in the U.S. and EU, and I believe a big part of this is because we haven’t yet required verification platforms to be liable for their system errors.

Within the act that’s a drafting problem. It can be improved with the simple addition of biometric verification into Annex III. 

Europe has made a real start on the deepfakes that want to be seen. The ones that don’t are still out there, working away in the space that this law leaves behind.