Explore our new Cybersecurity Terms Glossary, explained by our experts.
Cybersecurity Terms Glossary
An adversarial example is an input altered just enough to make a machine learning model misread it while a person sees nothing unusual, from pixel noise to printed patches worn on a face.

What Are Adversarial Examples?
An adversarial example is an input that has been deliberately altered so that a machine learning model misreads it, while a human observer notices nothing wrong. The change is usually small: a shift in a few pixel values, a substituted character, or a quiet layer of noise under an audio clip. What makes the failure awkward is that the model’s confidence often rises rather than falls, so there is no low score to trigger a fallback or a manual review.
A classifier divides a very high-dimensional input space into regions separated by decision boundaries, and most genuine inputs sit closer to a boundary than intuition suggests. If an attacker can compute gradients, they know exactly which direction pushes an input across the nearest boundary, and the distance to travel is measured in fractions of a pixel value. Ian Goodfellow’s 2014 analysis attributed the effect to the broadly linear behaviour of deep networks in high dimensions rather than to overfitting, which explains an uncomfortable property: a sample crafted against one model frequently fools a different model trained on different data for the same task. That transferability is what turns a laboratory curiosity into something usable against a system whose internals the attacker has never seen.
Researchers at Carnegie Mellon showed in 2016 that printed eyeglass frames with a carefully generated pattern let a wearer be recognised as a specific other person by a face recognition system. A Google team published the adversarial patch in 2017: a sticker that, placed beside any object, made classifiers report the object in the sticker instead. Nicholas Carlini and David Wagner demonstrated in 2018 that any audio waveform can be altered almost imperceptibly so that a speech recognition model transcribes an attacker-chosen sentence. Road-sign experiments in the same period established that weathered-looking stickers can change how a vision model reads a sign. Security products are affected too: machine learning malware classifiers have repeatedly been evaded by appending benign-looking bytes to a sample, which is one reason vendors keep signature and behavioural engines alongside their models.
Adversarial training, which adds generated attack samples to the training set, remains the most reliable mitigation, but it is expensive, trades away some clean accuracy, and hardens a model mainly against the attack family used to train it. Many published defences have been short-lived. Anish Athalye and colleagues showed in 2018 that most of them relied on obfuscated gradients and broke under adapted attacks. Certified approaches such as randomised smoothing give a provable guarantee, though only within a small perturbation radius. The practical conclusion is to treat model output as one signal rather than as a verdict: rate-limit and authenticate inference APIs, return labels instead of full confidence vectors, keep ensembles and non-ML checks in the decision path, and require human confirmation before anything irreversible happens.
The exposure sits wherever a model makes a decision that someone benefits from reversing: biometric onboarding in financial services, automated fraud scoring, number plate recognition, content moderation, and clinical imaging triage. The NCSC and CISA set out expectations for this in the Guidelines for Secure AI System Development, published in November 2023, and the Department for Science, Innovation and Technology followed with its code of practice for securing AI systems in January 2025, later issued as the technical specification ETSI TS 104 223. For attack taxonomy and test planning, NIST AI 100-2 and MITRE ATLAS are the two references worth keeping open. Evasion testing belongs in the same programme as your adversarial machine learning work and your wider AI security controls. At Privia Security, we include model evasion and abuse cases in our AI Penetration Test Services.
No. The same principle applies to audio, text, network traffic, and tabular data. Speech models can be driven to transcribe chosen phrases, text classifiers can be flipped by homoglyphs or invisible characters, and fraud or intrusion detection models can be evaded by shifting feature values within plausible ranges. Images are simply where the effect is easiest to demonstrate visually.
Yes. Two routes work without the weights. In a transfer attack, the attacker builds a substitute model on public data and reuses the examples it produces. In a query-based attack, they send thousands of inputs to your API and infer the decision boundary from the responses. Rate limiting, authentication, and returning coarse outputs all raise the cost of the second route.
It reduces it rather than solving it. A model trained on Projected Gradient Descent examples resists that family of attacks well, but accuracy on clean data usually drops, training costs rise sharply, and new attack types often find a way through. Treat it as one layer alongside output minimisation, anomaly monitoring, and a human check before consequential decisions.
Start by listing the decisions where a wrong answer has real consequences, then generate attacks against those paths with libraries such as the Adversarial Robustness Toolbox or Foolbox, in both white-box and query-only modes. Measure how many queries an attacker needs and what happens downstream when the model is wrong. MITRE ATLAS is useful for mapping the techniques you tested against real adversary behaviour.
You May Be Interested In These
We use a few cookies that are needed to make this website work, and they are always on. We would also like to set analytics cookies, to see which pages are useful; functional cookies, so that our live chat works; and advertising cookies, to measure our campaigns. We will not set any of those until you say yes, and you can change your mind at any time from the Cookie Settings link in the footer.
Choose the cookies you are happy for us to use. Strictly necessary cookies are always on; everything else is off until you turn it on.
These cookies keep the website working: they hold your security checks, your form submissions and the choice you make here. The site cannot run without them, so they are always on.
These let our live chat open and remember your conversation, and let the newsletter sign-up work. Switch them off and the chat window will not appear.
These tell us which pages are read and how people find us, so that we can improve the site. The reports are about groups of visitors, never about you personally.
These let us measure our advertising and show you our adverts on other websites, such as LinkedIn, Google and Meta. Switch them off and you will still see adverts, but they will be less relevant.