Explore our new Cybersecurity Terms Glossary, explained by our experts.
Cybersecurity Terms Glossary
Hash sets let an examiner remove millions of known-good files from a disk image and flag known-bad ones instantly, turning an unreviewable volume of data into a manageable set of candidates.

What Are Hash Sets and Known File Filtering?
A modern Windows installation contains well over half a million files before anyone saves a document. No examiner reviews them. Known file filtering removes them arithmetically: hash every file on the image, compare each value against a list of hashes known to belong to standard software, and exclude the matches. What remains is the material that is specific to this machine and this user, which is usually a few per cent of the original count.
Filtering works both ways, and the distinction matters when you design a workflow.
Known-good filtering, often called de-NISTing after the reference set that made it routine, is about reduction. The National Software Reference Library publishes the Reference Data Set, a continually updated collection of hashes for commercial, open-source, and operating system files, distributed by the National Institute of Standards and Technology. Excluding those leaves user documents, downloaded binaries, and anything the attacker brought with them.
Known-bad matching is about detection. Hash sets of malware samples, hacking tools, and previously seen indicators flag files the moment they are ingested. In an intrusion investigation these lists come from threat intelligence feeds, from previous cases, and from the indicators produced during the current one — hash a web shell found on one server and you can sweep the estate for it in minutes.
A cryptographic hash changes completely if one byte changes. That property is what makes it reliable and also what makes it brittle:
MD5 remains widespread in hash sets, and although collisions are well documented, that matters more for integrity claims than for filtering. For evidential work, record SHA-256 — see hashing for why the algorithm choice has consequences beyond convenience.
Similarity digests solve a different problem: finding files that are close rather than identical. ssdeep computes context-triggered piecewise hashes and returns a similarity score, which is effective at clustering variants of the same document or binary. TLSH and sdhash take different approaches to the same goal. In malware work, imphash hashes the import table of a portable executable, grouping samples compiled from the same source regardless of packing, and is often the fastest way to connect two apparently unrelated binaries to one toolkit.
Build and maintain your own sets alongside the public ones. A golden image hash set for each standard corporate build removes far more noise than a generic reference list, because it covers your line-of-business applications. Keep a case-specific known-bad set and reuse it on every related host. Record which sets were applied, with their versions and dates, in the examination log — if a file was excluded because it matched a known-good list, that decision is part of your methodology and an opposing expert is entitled to test it.
One caution for UK practice. Filtering against known-bad sets of illegal imagery is a law enforcement function, and the relevant national database is not available to private examiners. If an examination unexpectedly reveals indecent images of children, work stops and the matter goes to the police immediately; continuing to process the material is itself an offence, and no commercial instruction overrides that.
At Privia Security, we apply build-specific and case-specific hash sets at ingestion during our digital forensic incident response services, which is usually what turns a multi-terabyte collection into a review that finishes this week.
Removing files from a forensic review by matching their hashes against the Reference Data Set published by the National Software Reference Library, which catalogues hashes of known commercial and open-source software. The practice eliminates operating system and application files that no examiner needs to look at, typically cutting the review population by an order of magnitude before analysis begins.
No. Reference sets cannot cover bespoke applications, regional software, recent releases, or anything a user created. An unmatched file is simply unknown and needs triage by other means, such as its location, signature status, entropy, and execution artefacts. Treating unknown as malicious generates far more work than filtering saved.
When you expect variants rather than identical copies: clustering malware samples that share code, finding near-duplicate documents across custodians, or matching a modified tool against a known original. Tools such as ssdeep and TLSH return similarity scores instead of a binary match, and imphash groups executables by import table, which cuts through packing.
For filtering, yes, and most published sets still include it. For evidential integrity claims, prefer SHA-256, since deliberate MD5 collisions are demonstrable and the argument is easier to avoid than to win in cross-examination. Many tools compute both in a single pass, so recording the stronger value costs nothing in practice.
You May Be Interested In These
We use a few cookies that are needed to make this website work, and they are always on. We would also like to set analytics cookies, to see which pages are useful; functional cookies, so that our live chat works; and advertising cookies, to measure our campaigns. We will not set any of those until you say yes, and you can change your mind at any time from the Cookie Settings link in the footer.
Choose the cookies you are happy for us to use. Strictly necessary cookies are always on; everything else is off until you turn it on.
These cookies keep the website working: they hold your security checks, your form submissions and the choice you make here. The site cannot run without them, so they are always on.
These let our live chat open and remember your conversation, and let the newsletter sign-up work. Switch them off and the chat window will not appear.
These tell us which pages are read and how people find us, so that we can improve the site. The reports are about groups of visitors, never about you personally.
These let us measure our advertising and show you our adverts on other websites, such as LinkedIn, Google and Meta. Switch them off and you will still see adverts, but they will be less relevant.