Explore our new Cybersecurity Terms Glossary, explained by our experts.
Cybersecurity Terms Glossary
Adversarial machine learning is the study and practice of attacking AI models—fooling, poisoning, stealing, or interrogating them—and of building models that resist those attacks.

What Is Adversarial Machine Learning?
Adversarial machine learning is the field that studies how AI models can be deliberately manipulated—and how to defend them. An attacker might craft an input that a human sees as perfectly normal but that makes a model misclassify it, slip poisoned samples into training data, or query a model until they can copy it or extract the data it learned from. As organisations put machine learning into fraud detection, identity checks, and customer-facing chatbots, these attacks have moved from research papers into security assessments.
In 2013, researchers including Christian Szegedy showed that image classifiers could be fooled by tiny, carefully calculated changes to pixel values—changes invisible to a person. A year later, Ian Goodfellow and colleagues introduced the Fast Gradient Sign Method and the now-famous example of a panda photo that, after adding imperceptible noise, a network confidently labelled as a gibbon. Their key insight was that these “adversarial examples” are not rare bugs: they follow from how high-dimensional models draw decision boundaries, which is why they transfer between models trained on similar data.
| Attack | When it happens | What the attacker gets |
|---|---|---|
| Evasion | At inference time | A malicious input classified as benign, such as malware that slips past an ML-based detector |
| Poisoning | During training or fine-tuning | A degraded model, or a hidden backdoor triggered by a specific pattern |
| Model extraction | Through repeated API queries | A functional copy of a proprietary model |
| Privacy attacks | Through model outputs | Membership inference (was this record in the training set?) or reconstruction of training data |
Attacks are also described as white-box, where the attacker knows the model’s architecture and weights, or black-box, where they can only send inputs and observe outputs. Black-box attacks are slower but very realistic against commercial APIs.
Physical attacks prove the risk is not theoretical. In 2018, researchers showed that a few black and white stickers on a stop sign could make a vision model read it as a speed limit sign. In 2019, Tencent’s Keen Security Lab demonstrated that small stickers on a road surface could steer a Tesla’s lane-keeping system towards the wrong lane. Poisoning has a famous public example too: in 2016 Microsoft took its Tay chatbot offline within a day after users deliberately fed it offensive content that it learned to repeat.
For most organisations, the more pressing scenarios are closer to home: fraudsters testing which transaction patterns slip past a scoring model, document forgers probing an identity verification service, or synthetic media used against biometric checks, as covered in our deepfake entry.
Generative AI has widened the attack surface considerably. Prompt injection—hiding instructions in user input, a web page, or a document so the model ignores its original instructions—is ranked first (LLM01) in the OWASP Top 10 for LLM Applications 2025, and data and model poisoning appears as LLM04. When a model can call tools, send email, or query internal systems, a successful injection becomes far more than an embarrassing reply: it can turn into data exfiltration or unauthorised actions taken with the application’s privileges.
NIST’s AI 100-2 report, first published in January 2024 and updated in 2025, provides a shared taxonomy of attacks and mitigations, while MITRE ATLAS catalogues real adversary techniques against AI systems in the style of ATT&CK. In the UK, the NCSC co-published the Guidelines for Secure AI System Development in November 2023, and the Department for Science, Innovation and Technology released a Code of Practice for the Cyber Security of AI in January 2025. Where AI-enabled applications sit on your perimeter, we recommend including them in the scope of your next penetration test rather than treating them as a separate, untested layer.
No. Prompt injection is one specific attack against large language models, in which instructions hidden in input override the system’s intended behaviour. Adversarial machine learning is the wider field, covering evasion attacks on image and malware classifiers, training data poisoning, model theft, and privacy attacks as well as attacks on language models.
Yes, although your responsibilities shift. You may not control the model’s training, but you control what data you send it, what tools and systems it can reach, and how its output is used. Most real incidents with third-party models come from integration choices, such as giving a chatbot broad access to internal documents or letting it act without human approval.
Not with current techniques. Defences such as adversarial training and input filtering raise the cost of attacks but rarely eliminate them, and new attack methods regularly bypass published defences. The practical approach is defence in depth: limit what a model can do, monitor how it is used, validate its outputs, and test it adversarially on a regular basis.
Testing combines conventional application security work, such as authentication, access control, and API abuse, with AI-specific techniques: crafted adversarial inputs, prompt injection attempts, checks for training data leakage, and probing for unsafe tool use. Frameworks such as MITRE ATLAS and the OWASP Top 10 for LLM Applications help structure the scope.
You May Be Interested In These
We use a few cookies that are needed to make this website work, and they are always on. We would also like to set analytics cookies, to see which pages are useful; functional cookies, so that our live chat works; and advertising cookies, to measure our campaigns. We will not set any of those until you say yes, and you can change your mind at any time from the Cookie Settings link in the footer.
Choose the cookies you are happy for us to use. Strictly necessary cookies are always on; everything else is off until you turn it on.
These cookies keep the website working: they hold your security checks, your form submissions and the choice you make here. The site cannot run without them, so they are always on.
These let our live chat open and remember your conversation, and let the newsletter sign-up work. Switch them off and the chat window will not appear.
These tell us which pages are read and how people find us, so that we can improve the site. The reports are about groups of visitors, never about you personally.
These let us measure our advertising and show you our adverts on other websites, such as LinkedIn, Google and Meta. Switch them off and you will still see adverts, but they will be less relevant.