Explore our new Cybersecurity Terms Glossary, explained by our experts.

View the Glossary View the Glossary
PRIVIA

Cybersecurity Terms Glossary

What Are Hash Sets and Known File Filtering?

Hash sets let an examiner remove millions of known-good files from a disk image and flag known-bad ones instantly, turning an unreviewable volume of data into a manageable set of candidates.

Illustration for the cybersecurity term Hash Sets and Known File Filtering
Home / Blog / Cybersecurity Terms / What Are Hash Sets and Known File Filtering?
17 November 2026

What Are Hash Sets and Known File Filtering?

What Are Hash Sets and Known File Filtering?

A modern Windows installation contains well over half a million files before anyone saves a document. No examiner reviews them. Known file filtering removes them arithmetically: hash every file on the image, compare each value against a list of hashes known to belong to standard software, and exclude the matches. What remains is the material that is specific to this machine and this user, which is usually a few per cent of the original count.

The Two Directions

Filtering works both ways, and the distinction matters when you design a workflow.

Known-good filtering, often called de-NISTing after the reference set that made it routine, is about reduction. The National Software Reference Library publishes the Reference Data Set, a continually updated collection of hashes for commercial, open-source, and operating system files, distributed by the National Institute of Standards and Technology. Excluding those leaves user documents, downloaded binaries, and anything the attacker brought with them.

Known-bad matching is about detection. Hash sets of malware samples, hacking tools, and previously seen indicators flag files the moment they are ingested. In an intrusion investigation these lists come from threat intelligence feeds, from previous cases, and from the indicators produced during the current one — hash a web shell found on one server and you can sweep the estate for it in minutes.

Why Exact Hashing Is Not Enough

A cryptographic hash changes completely if one byte changes. That property is what makes it reliable and also what makes it brittle:

MD5 remains widespread in hash sets, and although collisions are well documented, that matters more for integrity claims than for filtering. For evidential work, record SHA-256 — see hashing for why the algorithm choice has consequences beyond convenience.

Fuzzy Hashing Fills the Gap

Similarity digests solve a different problem: finding files that are close rather than identical. ssdeep computes context-triggered piecewise hashes and returns a similarity score, which is effective at clustering variants of the same document or binary. TLSH and sdhash take different approaches to the same goal. In malware work, imphash hashes the import table of a portable executable, grouping samples compiled from the same source regardless of packing, and is often the fastest way to connect two apparently unrelated binaries to one toolkit.

Running It Properly

Build and maintain your own sets alongside the public ones. A golden image hash set for each standard corporate build removes far more noise than a generic reference list, because it covers your line-of-business applications. Keep a case-specific known-bad set and reuse it on every related host. Record which sets were applied, with their versions and dates, in the examination log — if a file was excluded because it matched a known-good list, that decision is part of your methodology and an opposing expert is entitled to test it.

One caution for UK practice. Filtering against known-bad sets of illegal imagery is a law enforcement function, and the relevant national database is not available to private examiners. If an examination unexpectedly reveals indecent images of children, work stops and the matter goes to the police immediately; continuing to process the material is itself an offence, and no commercial instruction overrides that.

At Privia Security, we apply build-specific and case-specific hash sets at ingestion during our digital forensic incident response services, which is usually what turns a multi-terabyte collection into a review that finishes this week.

FAQ – Frequently Asked Questions

What Is De-NISTing?

Removing files from a forensic review by matching their hashes against the Reference Data Set published by the National Software Reference Library, which catalogues hashes of known commercial and open-source software. The practice eliminates operating system and application files that no examiner needs to look at, typically cutting the review population by an order of magnitude before analysis begins.

Does a File That Is Not in a Hash Set Mean It Is Suspicious?

No. Reference sets cannot cover bespoke applications, regional software, recent releases, or anything a user created. An unmatched file is simply unknown and needs triage by other means, such as its location, signature status, entropy, and execution artefacts. Treating unknown as malicious generates far more work than filtering saved.

When Should I Use Fuzzy Hashing?

When you expect variants rather than identical copies: clustering malware samples that share code, finding near-duplicate documents across custodians, or matching a modified tool against a known original. Tools such as ssdeep and TLSH return similarity scores instead of a binary match, and imphash groups executables by import table, which cuts through packing.

Is MD5 Still Acceptable for Hash Sets?

For filtering, yes, and most published sets still include it. For evidential integrity claims, prefer SHA-256, since deliberate MD5 collisions are demonstrable and the argument is easier to avoid than to win in cross-examination. Many tools compute both in a single pass, so recording the stronger value costs nothing in practice.

Related Terms

You May Be Interested In These

What Is a SOAR Playbook?

Cybersecurity Terms

17 November 2026

What Is a SOAR Playbook?

Read More
What Is the SSH Protocol?

Cybersecurity Terms

17 November 2026

What Is the SSH Protocol?

Read More