Explore our new Cybersecurity Terms Glossary, explained by our experts.

View the Glossary View the Glossary
PRIVIA

Cybersecurity Terms Glossary

What Is AI Hallucination?

An AI hallucination is a confident, fluent, and false output from a generative model. The security problem is not the error itself but the systems and people that act on it.

Illustration for the cybersecurity term AI Hallucination
Home / Blog / Cybersecurity Terms / What Is AI Hallucination?
17 November 2026

What Is AI Hallucination?

What Is AI Hallucination?

A hallucination is output that a generative model presents with complete confidence and that is simply untrue: an invented court case, a function that does not exist in the library, a policy clause nobody wrote, a citation with a plausible author and a fabricated page number. The model is not lying in any meaningful sense. It is producing the most statistically likely continuation of the text so far, and fluency is not the same thing as accuracy.

Why Models Invent Things

Language models learn the shape of correct answers, not a lookup table of facts. When the training data is thin on a topic, when the question presumes something false, or when the decoding settings favour variety, the model fills the gap with something that fits the pattern. Several design choices make this worse: fine-tuning that rewards helpful-sounding answers over admissions of ignorance, long contexts in which details from different sources blur together, and retrieval systems that return weakly related documents the model then treats as authoritative. Hallucination rates fall with grounding and better prompting, but no current technique removes the behaviour, which is why every serious deployment is built around the assumption that some answers will be wrong.

When a Wrong Answer Becomes a Security Problem

Three categories matter to a security team.

Hallucination as an Attack Surface

Attackers do not only exploit spontaneous errors; they induce them. A loaded premise in a prompt (“summarise the exemption in clause 14 of the policy”) invites a model to confirm something that does not exist. Poisoned retrieval content pushes confident falsehoods into grounded systems, which is one reason data poisoning and hallucination are connected. The OWASP Top 10 for LLM Applications 2025 lists misinformation as LLM09 and emphasises the compounding risk of over-reliance, where users stop checking because the system is usually right.

Reducing the Damage

Grounding is the first lever: retrieval-augmented generation with citations that resolve to real documents, so a reader can check the source rather than trust the paraphrase. The second is constrained scope—a model answering from an approved knowledge base fabricates less than one answering from memory. The third is verification in code, not in guidance: resolve every package name against your internal registry before installation, validate generated configuration against a schema, and run generated code through the same review and scanning gates as any other contribution, as set out in our note on AI-generated code. Finally, design the interface so uncertainty is visible: an answer with no supporting source should look different from one with three.

Policy and Accountability

In the UK, the organisation deploying the system carries the consequences. The ICO expects accuracy to be addressed where personal data is involved, and professional regulators from the Solicitors Regulation Authority to the Financial Conduct Authority have made clear that AI assistance does not transfer responsibility for the output. An AI usage policy should therefore name the tasks where unverified model output may never be used, and training should cover how to check rather than only what is forbidden. At Privia Security, our consultancy team helps organisations set those boundaries before an AI assistant reaches customers.

FAQ – Frequently Asked Questions

Does Retrieval-Augmented Generation Eliminate Hallucination?

No, it reduces it. Grounding the model in retrieved documents cuts invented facts substantially, but the model can still misread a source, merge two documents, or answer confidently when retrieval returns nothing relevant. Citations help only if they resolve to a real passage a reader can check, so verify that your interface links to the source rather than printing a reference the model generated.

What Is Slopsquatting?

It is the practice of registering package names that AI coding assistants frequently invent, so that a developer who copies the suggestion installs attacker-controlled code. It works because hallucinated names are repeatable rather than random. The defence is simple and mechanical: resolve dependencies through an internal proxy or allow-list, and fail the build when a package is unknown.

Can We Measure How Often Our System Hallucinates?

Yes, with an evaluation set. Build a few hundred representative questions with verified answers from your own domain, run them on every model or prompt change, and score factual accuracy, citation validity, and the rate at which the system correctly declines. Track the numbers over time; a model upgrade that improves fluency can quietly make grounding worse.

Who Is Liable If Our Chatbot Gives a Customer Wrong Information?

In practice, the organisation running the chatbot. The Air Canada tribunal decision in February 2024 rejected the idea that a chatbot is a separate party, and UK consumer protection and financial promotion rules point the same way. Treat published model output as a statement by your business, with the review and record-keeping that implies.

Related Terms

You May Be Interested In These

What Is SNMP Security?

Cybersecurity Terms

17 November 2026

What Is SNMP Security?

Read More
What Is Trivy?

Cybersecurity Terms

17 November 2026

What Is Trivy?

Read More
What Is Certificate Transparency?

Cybersecurity Terms

17 November 2026

What Is Certificate Transparency?

Read More