Invisible attacks on AI

How do adversarial attacks against AI models work?

In 2014, Christian Szegedy and his research team published results that remain one of the most frequently cited demonstrations of AI model vulnerabilities. By adding a minimal, carefully calculated amount of noise to an image, they were able to alter the classifier’s decision while leaving the image visually unchanged to humans.

Such modified inputs are known as adversarial examples. 

The mechanism is surprisingly straightforward to formalize. Consider a machine learning model that correctly classifies an input x. An attacker seeks the smallest possible perturbation δ such that the model misclassifies x + δ, while δ remains imperceptible to human observers.

Valid Input → + Perturbation δ → AI Model → Incorrect Decision

Example:

• ECG signal, classified as “healthy”
• Perturbation satisfying ‖δ‖ ≤ ε and visually undetectable
• False classification as “ill”

Business and Legal Impact

• Incorrect diagnosis or automated decision
• Financial losses and loss of trust
• Potential non-compliance with Article 15 of the AI Act

The crucial difference between an adversarial attack and an ordinary model error lies in intent. This is not a random statistical mistake but a deliberate manipulation that exploits the way neural networks calculate gradients. AI models do not “see” the world the way humans do, and attackers understand this very well. 

Why is AI security now a business issue?

When machine learning models were primarily used to recommend movies, the consequences of errors were relatively minor. Today, similar architectures support decision-making in healthcare, biometrics, critical infrastructure, and anti-fraud systems.

In such environments, adversarially induced misclassifications can lead to:

• Incorrect ECG diagnoses
• Unauthorized access through biometric systems
• Fraudulent transactions being approved
• Manipulated readings from industrial sensors

As a result, adversarial vulnerability is no longer solely a data science concern. It becomes an organizational risk with an owner, likelihood, impact assessment, and required controls. In other words, it belongs in the organization’s AI risk register. 

This is where the regulatory dimension enters the discussion. The European Union’s AI Act (Regulation (EU) 2024/1689) entered into force on August 1, 2024, and is being applied progressively. One thing is already clear: robustness against attacks is becoming both a key security requirement and a legal obligation for AI systems. 

The AI Act and AI system robustness

The AI Act goes beyond general references to “security.” Article 15(1) requires high-risk AI systems to achieve and maintain an appropriate level of accuracy, robustness, and cybersecurity throughout their lifecycle.  Even more significant is Article 15(5), which explicitly identifies threats addressed in this research.

The provision requires organizations to implement measures capable of preventing, detecting, and responding to:

• Data poisoning attacks
• Manipulation of pretrained components (model poisoning)
• Inputs specifically designed to trigger incorrect model behavior, namely adversarial examples and model evasion attacks
• Attacks targeting model confidentiality and vulnerabilities

It is rare for legislation to describe attack techniques using the same terminology as scientific literature. For organizations, the implication is clear: “we were unaware” is no longer a viable defense.   

How should organizations manage AI risk?

The following simplified matrix can be incorporated directly into an organizational AI risk register.

Based on my observations, organizations frequently make three mistakes when addressing adversarial AI risks:

1. Treating AI models like conventional software: Traditional security testing methods, such as API penetration testing and vulnerability scanning, will not detect vulnerabilities embedded in model gradients.

2. Lack of clear risk ownership: Without assigning accountability to roles such as the CISO, Security Architect, or Data Manager, responsibility remains fragmented across teams.

3. Delaying action until regulations fully apply: Just as technical debt accumulates over time, so does what may be called “robustness debt.” Retrofitting security controls into production models is typically far more expensive than designing them from the outset. 

It is also worthwhile to align these efforts with recognized frameworks.

The NIST AI Risk Management Framework (AI RMF 1.0) structures AI governance around four core functions: Govern, Map, Measure, and Manage. Within the framework, the Measure function explicitly requires organizations to evaluate and document AI system security and robustness.

This creates a natural bridge between regulatory requirements under the AI Act and practical engineering implementation. 

What do adversarial attacks mean for businesses?

Adversarial attacks represent a measurable risk for any organization deploying AI systems in business-critical processes. The AI Act, particularly Article 15(5), explicitly recognizes these threats and requires active defensive measures. Meanwhile, the NIST AI RMF provides a structured approach for documenting and managing that risk. The good news is that improving AI robustness does not necessarily require large-scale or prohibitively expensive initiatives. In the next part of this series, I will demonstrate how relatively simple techniques can significantly reduce the susceptibility of AI systems to adversarial attacks.

---

In Part 2 of 4, I will break down a practical adversarial attack step by step and explain why the most popular “heavyweight” defense strategy is not always the most cost-effective option, as well as what organizations can implement instead.  

AUTHOR:
Piotr Hawryło is a Software Team Leader at ALTEN with more than 10 years of experience in embedded systems development. He holds an MBA in Cybersecurity from the Military University of Technology and has participated in machine learning projects, including initiatives related to exoplanet detection. His areas of expertise include software engineering, technical team leadership, and emerging technologies.