Explore our new Cybersecurity Terms Glossary, explained by our experts.

View the Glossary View the Glossary
PRIVIA

Cybersecurity Terms Glossary

What Is Adversarial Machine Learning?

Adversarial machine learning is the study and practice of attacking AI models—fooling, poisoning, stealing, or interrogating them—and of building models that resist those attacks.

Illustration for the cybersecurity term Adversarial Machine Learning
Home / Blog / Cybersecurity Terms / What Is Adversarial Machine Learning?
17 November 2026

What Is Adversarial Machine Learning?

What Is Adversarial Machine Learning?

Adversarial machine learning is the field that studies how AI models can be deliberately manipulated—and how to defend them. An attacker might craft an input that a human sees as perfectly normal but that makes a model misclassify it, slip poisoned samples into training data, or query a model until they can copy it or extract the data it learned from. As organisations put machine learning into fraud detection, identity checks, and customer-facing chatbots, these attacks have moved from research papers into security assessments.

Where the Field Began

In 2013, researchers including Christian Szegedy showed that image classifiers could be fooled by tiny, carefully calculated changes to pixel values—changes invisible to a person. A year later, Ian Goodfellow and colleagues introduced the Fast Gradient Sign Method and the now-famous example of a panda photo that, after adding imperceptible noise, a network confidently labelled as a gibbon. Their key insight was that these “adversarial examples” are not rare bugs: they follow from how high-dimensional models draw decision boundaries, which is why they transfer between models trained on similar data.

The Main Attack Families

AttackWhen it happensWhat the attacker gets
EvasionAt inference timeA malicious input classified as benign, such as malware that slips past an ML-based detector
PoisoningDuring training or fine-tuningA degraded model, or a hidden backdoor triggered by a specific pattern
Model extractionThrough repeated API queriesA functional copy of a proprietary model
Privacy attacksThrough model outputsMembership inference (was this record in the training set?) or reconstruction of training data

Attacks are also described as white-box, where the attacker knows the model’s architecture and weights, or black-box, where they can only send inputs and observe outputs. Black-box attacks are slower but very realistic against commercial APIs.

From the Lab to the Real World

Physical attacks prove the risk is not theoretical. In 2018, researchers showed that a few black and white stickers on a stop sign could make a vision model read it as a speed limit sign. In 2019, Tencent’s Keen Security Lab demonstrated that small stickers on a road surface could steer a Tesla’s lane-keeping system towards the wrong lane. Poisoning has a famous public example too: in 2016 Microsoft took its Tay chatbot offline within a day after users deliberately fed it offensive content that it learned to repeat.

For most organisations, the more pressing scenarios are closer to home: fraudsters testing which transaction patterns slip past a scoring model, document forgers probing an identity verification service, or synthetic media used against biometric checks, as covered in our deepfake entry.

Large Language Models Change the Picture

Generative AI has widened the attack surface considerably. Prompt injection—hiding instructions in user input, a web page, or a document so the model ignores its original instructions—is ranked first (LLM01) in the OWASP Top 10 for LLM Applications 2025, and data and model poisoning appears as LLM04. When a model can call tools, send email, or query internal systems, a successful injection becomes far more than an embarrassing reply: it can turn into data exfiltration or unauthorised actions taken with the application’s privileges.

Building More Resilient Models

Frameworks and UK Guidance

NIST’s AI 100-2 report, first published in January 2024 and updated in 2025, provides a shared taxonomy of attacks and mitigations, while MITRE ATLAS catalogues real adversary techniques against AI systems in the style of ATT&CK. In the UK, the NCSC co-published the Guidelines for Secure AI System Development in November 2023, and the Department for Science, Innovation and Technology released a Code of Practice for the Cyber Security of AI in January 2025. Where AI-enabled applications sit on your perimeter, we recommend including them in the scope of your next penetration test rather than treating them as a separate, untested layer.

FAQ – Frequently Asked Questions

Is Adversarial Machine Learning the Same as Prompt Injection?

No. Prompt injection is one specific attack against large language models, in which instructions hidden in input override the system’s intended behaviour. Adversarial machine learning is the wider field, covering evasion attacks on image and malware classifiers, training data poisoning, model theft, and privacy attacks as well as attacks on language models.

Do I Need to Worry About This if We Only Use Third-Party AI Services?

Yes, although your responsibilities shift. You may not control the model’s training, but you control what data you send it, what tools and systems it can reach, and how its output is used. Most real incidents with third-party models come from integration choices, such as giving a chatbot broad access to internal documents or letting it act without human approval.

Can Adversarial Attacks Be Completely Prevented?

Not with current techniques. Defences such as adversarial training and input filtering raise the cost of attacks but rarely eliminate them, and new attack methods regularly bypass published defences. The practical approach is defence in depth: limit what a model can do, monitor how it is used, validate its outputs, and test it adversarially on a regular basis.

How Is an AI System Security Tested?

Testing combines conventional application security work, such as authentication, access control, and API abuse, with AI-specific techniques: crafted adversarial inputs, prompt injection attempts, checks for training data leakage, and probing for unsafe tool use. Frameworks such as MITRE ATLAS and the OWASP Top 10 for LLM Applications help structure the scope.

Related Terms

You May Be Interested In These

What Is an AI Usage Policy?

Cybersecurity Terms

17 November 2026

What Is an AI Usage Policy?

Read More
What Are FIDO2 and WebAuthn?

Cybersecurity Terms

17 November 2026

What Are FIDO2 and WebAuthn?

Read More
What Is Path Traversal?

Cybersecurity Terms

17 November 2026

What Is Path Traversal?

Read More