Explore our new Cybersecurity Terms Glossary, explained by our experts.

View the Glossary View the Glossary
PRIVIA

Cybersecurity Terms Glossary

What Are Adversarial Examples?

An adversarial example is an input altered just enough to make a machine learning model misread it while a person sees nothing unusual, from pixel noise to printed patches worn on a face.

Illustration for the cybersecurity term Adversarial Examples
Home / Blog / Cybersecurity Terms / What Are Adversarial Examples?
17 November 2026

What Are Adversarial Examples?

What Are Adversarial Examples?

An adversarial example is an input that has been deliberately altered so that a machine learning model misreads it, while a human observer notices nothing wrong. The change is usually small: a shift in a few pixel values, a substituted character, or a quiet layer of noise under an audio clip. What makes the failure awkward is that the model’s confidence often rises rather than falls, so there is no low score to trigger a fallback or a manual review.

Why a Little Noise Goes a Long Way

A classifier divides a very high-dimensional input space into regions separated by decision boundaries, and most genuine inputs sit closer to a boundary than intuition suggests. If an attacker can compute gradients, they know exactly which direction pushes an input across the nearest boundary, and the distance to travel is measured in fractions of a pixel value. Ian Goodfellow’s 2014 analysis attributed the effect to the broadly linear behaviour of deep networks in high dimensions rather than to overfitting, which explains an uncomfortable property: a sample crafted against one model frequently fools a different model trained on different data for the same task. That transferability is what turns a laboratory curiosity into something usable against a system whose internals the attacker has never seen.

White-Box, Black-Box, and the Space Between

The Attacks That Left the Laboratory

Researchers at Carnegie Mellon showed in 2016 that printed eyeglass frames with a carefully generated pattern let a wearer be recognised as a specific other person by a face recognition system. A Google team published the adversarial patch in 2017: a sticker that, placed beside any object, made classifiers report the object in the sticker instead. Nicholas Carlini and David Wagner demonstrated in 2018 that any audio waveform can be altered almost imperceptibly so that a speech recognition model transcribes an attacker-chosen sentence. Road-sign experiments in the same period established that weathered-looking stickers can change how a vision model reads a sign. Security products are affected too: machine learning malware classifiers have repeatedly been evaded by appending benign-looking bytes to a sample, which is one reason vendors keep signature and behavioural engines alongside their models.

Defences and Where They Stop

Adversarial training, which adds generated attack samples to the training set, remains the most reliable mitigation, but it is expensive, trades away some clean accuracy, and hardens a model mainly against the attack family used to train it. Many published defences have been short-lived. Anish Athalye and colleagues showed in 2018 that most of them relied on obfuscated gradients and broke under adapted attacks. Certified approaches such as randomised smoothing give a provable guarantee, though only within a small perturbation radius. The practical conclusion is to treat model output as one signal rather than as a verdict: rate-limit and authenticate inference APIs, return labels instead of full confidence vectors, keep ensembles and non-ML checks in the decision path, and require human confirmation before anything irreversible happens.

What This Means for UK Organisations

The exposure sits wherever a model makes a decision that someone benefits from reversing: biometric onboarding in financial services, automated fraud scoring, number plate recognition, content moderation, and clinical imaging triage. The NCSC and CISA set out expectations for this in the Guidelines for Secure AI System Development, published in November 2023, and the Department for Science, Innovation and Technology followed with its code of practice for securing AI systems in January 2025, later issued as the technical specification ETSI TS 104 223. For attack taxonomy and test planning, NIST AI 100-2 and MITRE ATLAS are the two references worth keeping open. Evasion testing belongs in the same programme as your adversarial machine learning work and your wider AI security controls. At Privia Security, we include model evasion and abuse cases in our AI Penetration Test Services.

FAQ – Frequently Asked Questions

Are Adversarial Examples Only a Problem for Image Models?

No. The same principle applies to audio, text, network traffic, and tabular data. Speech models can be driven to transcribe chosen phrases, text classifiers can be flipped by homoglyphs or invisible characters, and fraud or intrusion detection models can be evaded by shifting feature values within plausible ranges. Images are simply where the effect is easiest to demonstrate visually.

Can an Attacker Craft These Without Access to Our Model?

Yes. Two routes work without the weights. In a transfer attack, the attacker builds a substitute model on public data and reuses the examples it produces. In a query-based attack, they send thousands of inputs to your API and infer the decision boundary from the responses. Rate limiting, authentication, and returning coarse outputs all raise the cost of the second route.

Does Adversarial Training Solve the Problem?

It reduces it rather than solving it. A model trained on Projected Gradient Descent examples resists that family of attacks well, but accuracy on clean data usually drops, training costs rise sharply, and new attack types often find a way through. Treat it as one layer alongside output minimisation, anomaly monitoring, and a human check before consequential decisions.

How Do We Test Our Own Models for This?

Start by listing the decisions where a wrong answer has real consequences, then generate attacks against those paths with libraries such as the Adversarial Robustness Toolbox or Foolbox, in both white-box and query-only modes. Measure how many queries an attacker needs and what happens downstream when the model is wrong. MITRE ATLAS is useful for mapping the techniques you tested against real adversary behaviour.

Related Terms

You May Be Interested In These

What Is Process Hollowing?

Cybersecurity Terms

17 November 2026

What Is Process Hollowing?

Read More
What Is a URL Address? How Does It Work?

Cybersecurity Terms

30 March 2021

What Is a URL Address? How Does It Work?

Read More
What Is Living off the Land (LotL)?

Cybersecurity Terms

17 November 2026

What Is Living off the Land (LotL)?

Read More