The massive rise of artificial intelligence (AI), especially after ChatGPT was released, brought along concerns about the latest security breach, one that even machines might not notice: adversarial attacks.
These threats turn AI against itself, feeding the model with wrong data. The most famous example to illustrate how they happen involves a panda.
Back in 2014, researchers at Google and New York University simulated adversarial attacks. They added noise to an image that a Google AI model correctly recognised as a panda. After the “attack”, the same model now would recognise it as a gibbon – even though, for human eyes, the animal was still clearly a panda.
Since you and I don’t usually have to distinguish pandas from gibbons very often in our daily lives, why are adversarial attacks such as the one we described so important?
According to Henrique Arcoverde, Director of Consulting and Software Engineering at Tempest Security Intelligence, hackers already use adversarial attacks for financial gain and even bankrupt companies.
Before moving forward, a brief explanation: AI models run several scenarios for a given task and generate a graph like the one below. The results will be placed over or under the decision boundary – a line separating right and wrong outcomes. Here’s where adversarial attacks take place.
“[Adversarial attacks] take results near the boundary and introduce some noise that does not change the semantics, the meaning of information – but changes its classification,” Arcoverde said in a presentation at the Brazilian conference Forum E-Commerce Brasil.
It might not sound terribly concerning, but try to think about it from this perspective: “AI is widely used for combating financial frauds and heavily used in e-commerce ecosystems,” Arcoverde pointed out.
Misclassifications have already impacted companies and users and are expected to become even more frequent. Arcoverde broke down eight scenarios in which adversarial attacks were used or could be used in real life.
Fraud detection
In this case, attacks change how AI models classify transactions, making the system block a valid online purchase or let a fraudulent order pass through. So far, the market has not seen a massive attack of such type, even though researchers have successfully simulated it in safe environments.
Content moderation
Here, the idea is to change what type of content the AI model understands as appropriate and what should be moderated – offensive comments and pornography, for example. Adversarial attacks in this category often use image recognition and language processing.
Spam and phishing
Hackers try to subvert the definition of spam and phishing that an AI model uses. The goal is to bypass security systems and retrieve sensitive data.
Reviews and recommendations
By fooling AI models that detect fake reviews on product pages, one can create click farms to boost items’ ratings or make false, positive comments about a given company, for example. Arcoverde expects this type of attack to grow in the future.
Sentiment analysis
Brands keep track of what consumers say about them online: are users enjoying the software’s latest update? How are they reacting to a marketing campaign? Adversarial attacks, however, can trick the AI model into classifying sentiments in a different way – for the better or the worse.
Dynamic pricing
One of the most common applications of AI in commerce is to update prices depending on the market – if competitors are lowering their prices, you probably should do the same, and the opposite is also true.
According to Arcoverde, a relatively big company was recently subject to an attack that would inflate prices in the market. The AI pricing model did not detect the threat and automatically increased the products’ cost in response. The problem was that its competitors were not making the items more expensive, resulting in a significant financial loss.
Model leak
What about using adversarial attacks to “steal” the actual AI model? That is already happening: depending on how the generative AI was built, one can retrieve data about how the software was trained.
Identity theft
With the introduction of liveness – a way to authenticate someone’s identity in real-time by asking the user to move the camera for a selfie, for instance – adversarial attacks use AI to steal sensitive data about the user and perform actions on their behalf illegally.
Journalist since eight years old, when I would read the newspaper out loud and pretend it was a radio show. Based in São Paulo, I have worked for Brazilian websites as reporter and editor before joining 6GWorld
