Explore our new Cybersecurity Terms Glossary, explained by our experts.
Cybersecurity Terms Glossary
An AI hallucination is a confident, fluent, and false output from a generative model. The security problem is not the error itself but the systems and people that act on it.

What Is AI Hallucination?
A hallucination is output that a generative model presents with complete confidence and that is simply untrue: an invented court case, a function that does not exist in the library, a policy clause nobody wrote, a citation with a plausible author and a fabricated page number. The model is not lying in any meaningful sense. It is producing the most statistically likely continuation of the text so far, and fluency is not the same thing as accuracy.
Language models learn the shape of correct answers, not a lookup table of facts. When the training data is thin on a topic, when the question presumes something false, or when the decoding settings favour variety, the model fills the gap with something that fits the pattern. Several design choices make this worse: fine-tuning that rewards helpful-sounding answers over admissions of ignorance, long contexts in which details from different sources blur together, and retrieval systems that return weakly related documents the model then treats as authoritative. Hallucination rates fall with grounding and better prompting, but no current technique removes the behaviour, which is why every serious deployment is built around the assumption that some answers will be wrong.
Three categories matter to a security team.
Attackers do not only exploit spontaneous errors; they induce them. A loaded premise in a prompt (“summarise the exemption in clause 14 of the policy”) invites a model to confirm something that does not exist. Poisoned retrieval content pushes confident falsehoods into grounded systems, which is one reason data poisoning and hallucination are connected. The OWASP Top 10 for LLM Applications 2025 lists misinformation as LLM09 and emphasises the compounding risk of over-reliance, where users stop checking because the system is usually right.
Grounding is the first lever: retrieval-augmented generation with citations that resolve to real documents, so a reader can check the source rather than trust the paraphrase. The second is constrained scope—a model answering from an approved knowledge base fabricates less than one answering from memory. The third is verification in code, not in guidance: resolve every package name against your internal registry before installation, validate generated configuration against a schema, and run generated code through the same review and scanning gates as any other contribution, as set out in our note on AI-generated code. Finally, design the interface so uncertainty is visible: an answer with no supporting source should look different from one with three.
In the UK, the organisation deploying the system carries the consequences. The ICO expects accuracy to be addressed where personal data is involved, and professional regulators from the Solicitors Regulation Authority to the Financial Conduct Authority have made clear that AI assistance does not transfer responsibility for the output. An AI usage policy should therefore name the tasks where unverified model output may never be used, and training should cover how to check rather than only what is forbidden. At Privia Security, our consultancy team helps organisations set those boundaries before an AI assistant reaches customers.
No, it reduces it. Grounding the model in retrieved documents cuts invented facts substantially, but the model can still misread a source, merge two documents, or answer confidently when retrieval returns nothing relevant. Citations help only if they resolve to a real passage a reader can check, so verify that your interface links to the source rather than printing a reference the model generated.
It is the practice of registering package names that AI coding assistants frequently invent, so that a developer who copies the suggestion installs attacker-controlled code. It works because hallucinated names are repeatable rather than random. The defence is simple and mechanical: resolve dependencies through an internal proxy or allow-list, and fail the build when a package is unknown.
Yes, with an evaluation set. Build a few hundred representative questions with verified answers from your own domain, run them on every model or prompt change, and score factual accuracy, citation validity, and the rate at which the system correctly declines. Track the numbers over time; a model upgrade that improves fluency can quietly make grounding worse.
In practice, the organisation running the chatbot. The Air Canada tribunal decision in February 2024 rejected the idea that a chatbot is a separate party, and UK consumer protection and financial promotion rules point the same way. Treat published model output as a statement by your business, with the review and record-keeping that implies.
You May Be Interested In These
We use a few cookies that are needed to make this website work, and they are always on. We would also like to set analytics cookies, to see which pages are useful; functional cookies, so that our live chat works; and advertising cookies, to measure our campaigns. We will not set any of those until you say yes, and you can change your mind at any time from the Cookie Settings link in the footer.
Choose the cookies you are happy for us to use. Strictly necessary cookies are always on; everything else is off until you turn it on.
These cookies keep the website working: they hold your security checks, your form submissions and the choice you make here. The site cannot run without them, so they are always on.
These let our live chat open and remember your conversation, and let the newsletter sign-up work. Switch them off and the chat window will not appear.
These tell us which pages are read and how people find us, so that we can improve the site. The reports are about groups of visitors, never about you personally.
These let us measure our advertising and show you our adverts on other websites, such as LinkedIn, Google and Meta. Switch them off and you will still see adverts, but they will be less relevant.