Explore our new Cybersecurity Terms Glossary, explained by our experts.

View the Glossary View the Glossary
PRIVIA

Cybersecurity Terms Glossary

What Is Model Extraction?

Model extraction rebuilds a functional copy of a proprietary model by querying its API, turning an expensive asset into something a competitor can reproduce for a fraction of the cost.

Illustration for the cybersecurity term Model Extraction
Home / Blog / Cybersecurity Terms / What Is Model Extraction?
17 November 2026

What Is Model Extraction?

What Is Model Extraction?

Model extraction, also called model stealing, is the reconstruction of a model you do not have access to by querying the interface you do have. The attacker sends inputs, records outputs, and trains their own model on those pairs until it behaves close enough to the original. Nothing is downloaded and no system is breached. The intellectual property walks out through the front door, one API call at a time.

The Original Result and What Changed

Florian Tramèr and colleagues demonstrated the attack at USENIX Security in 2016, recovering decision trees, logistic regression models, and small neural networks from commercial prediction APIs—in some cases with near-perfect fidelity after only a few thousand queries. Two things made the problem larger since. Models became far more expensive to train, so the asset being stolen is worth orders of magnitude more. And generative models made extraction easier in one respect: rich text output carries far more information per query than a class label, so an attacker can train a capable student model on a comparatively modest number of responses. Distillation from a commercial model’s outputs is now a well-documented route to producing a competitive model cheaply, and most provider terms of service prohibit it precisely because it works.

Extraction Variants

Why It Is More Than an IP Problem

A stolen copy removes the rate limits. Once an attacker holds a local model that approximates yours, they can search it offline for adversarial examples that transfer back to production, probe it for membership inference without being detected, and map its refusal boundaries to design a reliable jailbreak. This is why extraction is often an enabling step rather than the objective. For a fraud or abuse detection model, the consequence is immediate: an attacker with a local copy can tune their activity until it scores as benign, every time, before touching your systems.

Defences and Their Trade-Offs

No single control stops a patient attacker, but the economics can be shifted.

The UK Legal Position

Model weights are protected as confidential information and, depending on how they were produced and expressed, potentially by copyright and database right. Systematic extraction through an API in breach of terms of service may also engage the Computer Misuse Act 1990, since access exceeding authorisation is the statutory test rather than circumvention of a technical barrier. Practically, enforcement depends on evidence, which means logging query patterns and retaining them, and on being able to demonstrate that a suspected copy derives from your model—the argument watermarking is designed to support. NIST AI 100-2 treats extraction as a core attack class, and it appears in MITRE ATLAS as an adversary objective. At Privia Security, we assess inference endpoints for extraction exposure, rate limiting, and output minimisation as part of our AI Penetration Test Services.

FAQ – Frequently Asked Questions

How Many Queries Does Extraction Take?

It varies enormously with the model and the information returned per query. Small classifiers have been reconstructed in thousands of queries. For large generative models, a useful distilled copy can be produced from tens or hundreds of thousands of responses, which is affordable at public API prices. Returning less information per response is the most direct way to increase the number required.

Does Rate Limiting Stop It?

It raises the cost without removing the risk. A determined attacker spreads queries across many accounts, uses residential proxies, and extracts slowly over weeks. Combine rate limiting with authentication, behavioural detection of systematic query patterns, and output minimisation, which together make extraction expensive enough that most opportunists move on.

Can We Prove Someone Copied Our Model?

Only with preparation. Watermarking the model before release gives you a hidden signature to test for in a suspected copy, and retained query logs can evidence the extraction campaign itself. Without either, proving derivation from behavioural similarity alone is difficult, because two models trained on similar data legitimately behave similarly.

Is Distilling From a Commercial Model Legal?

Provider terms of service generally prohibit using outputs to train competing models, so it is at minimum a contractual breach. Depending on the circumstances it may also engage confidential information, copyright, or computer misuse claims in the UK. Organisations should be clear in their own AI usage policy that training on another provider’s outputs is not permitted without legal review.

Related Terms

You May Be Interested In These

What Is Number Porting Fraud?

Cybersecurity Terms

17 November 2026

What Is Number Porting Fraud?

Read More
What Is RPKI?

Cybersecurity Terms

17 November 2026

What Is RPKI?

Read More
What Is Evil-WinRM?

Cybersecurity Terms

17 November 2026

What Is Evil-WinRM?

Read More