Explore our new Cybersecurity Terms Glossary, explained by our experts.
Cybersecurity Terms Glossary
Model extraction rebuilds a functional copy of a proprietary model by querying its API, turning an expensive asset into something a competitor can reproduce for a fraction of the cost.

What Is Model Extraction?
Model extraction, also called model stealing, is the reconstruction of a model you do not have access to by querying the interface you do have. The attacker sends inputs, records outputs, and trains their own model on those pairs until it behaves close enough to the original. Nothing is downloaded and no system is breached. The intellectual property walks out through the front door, one API call at a time.
Florian Tramèr and colleagues demonstrated the attack at USENIX Security in 2016, recovering decision trees, logistic regression models, and small neural networks from commercial prediction APIs—in some cases with near-perfect fidelity after only a few thousand queries. Two things made the problem larger since. Models became far more expensive to train, so the asset being stolen is worth orders of magnitude more. And generative models made extraction easier in one respect: rich text output carries far more information per query than a class label, so an attacker can train a capable student model on a comparatively modest number of responses. Distillation from a commercial model’s outputs is now a well-documented route to producing a competitive model cheaply, and most provider terms of service prohibit it precisely because it works.
A stolen copy removes the rate limits. Once an attacker holds a local model that approximates yours, they can search it offline for adversarial examples that transfer back to production, probe it for membership inference without being detected, and map its refusal boundaries to design a reliable jailbreak. This is why extraction is often an enabling step rather than the objective. For a fraud or abuse detection model, the consequence is immediate: an attacker with a local copy can tune their activity until it scores as benign, every time, before touching your systems.
No single control stops a patient attacker, but the economics can be shifted.
Model weights are protected as confidential information and, depending on how they were produced and expressed, potentially by copyright and database right. Systematic extraction through an API in breach of terms of service may also engage the Computer Misuse Act 1990, since access exceeding authorisation is the statutory test rather than circumvention of a technical barrier. Practically, enforcement depends on evidence, which means logging query patterns and retaining them, and on being able to demonstrate that a suspected copy derives from your model—the argument watermarking is designed to support. NIST AI 100-2 treats extraction as a core attack class, and it appears in MITRE ATLAS as an adversary objective. At Privia Security, we assess inference endpoints for extraction exposure, rate limiting, and output minimisation as part of our AI Penetration Test Services.
It varies enormously with the model and the information returned per query. Small classifiers have been reconstructed in thousands of queries. For large generative models, a useful distilled copy can be produced from tens or hundreds of thousands of responses, which is affordable at public API prices. Returning less information per response is the most direct way to increase the number required.
It raises the cost without removing the risk. A determined attacker spreads queries across many accounts, uses residential proxies, and extracts slowly over weeks. Combine rate limiting with authentication, behavioural detection of systematic query patterns, and output minimisation, which together make extraction expensive enough that most opportunists move on.
Only with preparation. Watermarking the model before release gives you a hidden signature to test for in a suspected copy, and retained query logs can evidence the extraction campaign itself. Without either, proving derivation from behavioural similarity alone is difficult, because two models trained on similar data legitimately behave similarly.
Provider terms of service generally prohibit using outputs to train competing models, so it is at minimum a contractual breach. Depending on the circumstances it may also engage confidential information, copyright, or computer misuse claims in the UK. Organisations should be clear in their own AI usage policy that training on another provider’s outputs is not permitted without legal review.
You May Be Interested In These
We use a few cookies that are needed to make this website work, and they are always on. We would also like to set analytics cookies, to see which pages are useful; functional cookies, so that our live chat works; and advertising cookies, to measure our campaigns. We will not set any of those until you say yes, and you can change your mind at any time from the Cookie Settings link in the footer.
Choose the cookies you are happy for us to use. Strictly necessary cookies are always on; everything else is off until you turn it on.
These cookies keep the website working: they hold your security checks, your form submissions and the choice you make here. The site cannot run without them, so they are always on.
These let our live chat open and remember your conversation, and let the newsletter sign-up work. Switch them off and the chat window will not appear.
These tell us which pages are read and how people find us, so that we can improve the site. The reports are about groups of visitors, never about you personally.
These let us measure our advertising and show you our adverts on other websites, such as LinkedIn, Google and Meta. Switch them off and you will still see adverts, but they will be less relevant.