Explore our new Cybersecurity Terms Glossary, explained by our experts.
Cybersecurity Terms Glossary
An LLM jailbreak is a prompt or conversation designed to get a language model to ignore its safety training and produce output its provider intended it to refuse.

What Is an LLM Jailbreak?
A jailbreak is a way of talking to a language model that gets it to do something its safety training says it should refuse. No vulnerability is exploited in the software sense. The attacker simply finds a framing—a story, a hypothetical, a translation, a long sequence of small requests—in which the refusal behaviour does not fire. The name comes from mobile phone jailbreaking, and like that practice it has a large hobbyist community alongside its security relevance.
Safety behaviour is trained in, usually through reinforcement learning from human feedback, and training teaches patterns rather than rules. The model learns that requests resembling the refusal examples should be declined, so an attacker’s job is to construct a request that does not resemble them while still producing the content. Three structural facts keep this working: the input space of natural language is unbounded, helpfulness and caution are in tension so models are tuned towards being useful, and the model has no persistent notion of a policy it can consult. Safety training reduces how often these attacks succeed; it does not establish a boundary.
These terms overlap and are often used loosely. A jailbreak targets the model provider’s safety policy, and the person running it is usually the person who wants the output, so the immediate harm falls on the provider’s reputation and on third parties. Prompt injection targets the application’s instructions, and in its indirect form the attacker and the victim are different people. For a business deploying a model, the second is the more serious security problem, but jailbreaks still matter: a customer-facing assistant induced to produce abusive content, regulated financial advice, or instructions for self-harm is a reputational and regulatory incident regardless of who prompted it.
You cannot retrain a hosted provider’s model, so your controls sit around it. Apply guardrails on input and output, with an output classifier being the more valuable of the two because it judges what actually came back rather than guessing intent. Narrow the scope: an assistant restricted to your product documentation has fewer interesting places to be driven. Authenticate users so that abuse is attributable and rate-limited, since most automated jailbreak searches need many attempts. Log conversations well enough to investigate a complaint. And accept that a determined individual with unlimited attempts will eventually succeed, so plan the response—content takedown, account suspension, disclosure—before it happens rather than during.
Jailbreak resistance should be measured rather than assumed, which is a core part of AI red teaming: run a current corpus of techniques, report success rates per harm category, and re-run after every model or prompt change, because provider updates routinely close some paths and open others. The UK AI Security Institute publishes evaluation methodology that is useful as a reference even for organisations deploying rather than building models, and the NCSC and CISA Guidelines for Secure AI System Development expect adversarial testing before release. Providers of general-purpose models with systemic risk face an explicit testing duty under the EU AI Act. At Privia Security, we measure jailbreak resistance against current technique libraries in our AI Penetration Test Services.
The act of prompting a model in an unusual way is not in itself an offence, but it may breach the provider’s terms of service, and what you do with the output can certainly be unlawful. Producing or distributing illegal material, facilitating fraud, or using the output to attack a system all attract the usual criminal and civil consequences.
No deployed system has achieved that, and claims to the contrary have a short shelf life. Aim instead to raise the cost and limit the consequences: narrow the assistant’s scope, filter output, authenticate and rate-limit users, log everything, and ensure that a successful jailbreak produces embarrassing text rather than access to data or the ability to act.
Not by itself. A jailbreak changes what the model is willing to say. It becomes a security incident when the model can act—calling tools, querying internal data, sending messages—or when its output is passed unvalidated into another system. That is why least privilege and output validation matter more than trying to win the prompt arms race.
After every change to the model, system prompt, guardrail configuration, or connected tools, and on a regular cadence in between. Provider model updates are the most commonly missed trigger, because they happen without any change on your side and can alter refusal behaviour in both directions. Keep a regression suite so retesting is a scripted exercise rather than a project.
You May Be Interested In These
We use a few cookies that are needed to make this website work, and they are always on. We would also like to set analytics cookies, to see which pages are useful; functional cookies, so that our live chat works; and advertising cookies, to measure our campaigns. We will not set any of those until you say yes, and you can change your mind at any time from the Cookie Settings link in the footer.
Choose the cookies you are happy for us to use. Strictly necessary cookies are always on; everything else is off until you turn it on.
These cookies keep the website working: they hold your security checks, your form submissions and the choice you make here. The site cannot run without them, so they are always on.
These let our live chat open and remember your conversation, and let the newsletter sign-up work. Switch them off and the chat window will not appear.
These tell us which pages are read and how people find us, so that we can improve the site. The reports are about groups of visitors, never about you personally.
These let us measure our advertising and show you our adverts on other websites, such as LinkedIn, Google and Meta. Switch them off and you will still see adverts, but they will be less relevant.