Explore our new Cybersecurity Terms Glossary, explained by our experts.

View the Glossary View the Glossary
PRIVIA

Cybersecurity Terms Glossary

What Is a Security Data Lake?

A security data lake stores security telemetry in open formats on cheap object storage, so that years of data stay queryable at a cost that log-licensed SIEM pricing cannot match.

Illustration for the cybersecurity term Security Data Lake
Home / Blog / Cybersecurity Terms / What Is a Security Data Lake?
17 November 2026

What Is a Security Data Lake?

What Is a Security Data Lake?

Ask a detection team what they would investigate if storage were free, and the answer is always the same: further back. Dwell times are measured in months, threat intelligence arrives long after the infrastructure it names was active, and the single most useful question during an incident — has this ever happened before — is usually unanswerable because the logs expired. A security data lake is the architectural response to that problem.

It stores security telemetry in open file formats, typically Parquet organised as Apache Iceberg or Delta tables, on object storage such as Amazon S3 or Azure Data Lake Storage. Compute is applied at query time rather than at ingestion, so the cost of keeping data and the cost of analysing it are decoupled. That single property is what makes several years of retention affordable.

How It Differs From a SIEM

The comparison people reach for is misleading, because the two are not competing for the same job.

SIEMSecurity data lake
Primary purposeReal-time detection, alerting, case managementLong retention, large-scale analysis, retrospective search
SchemaApplied at ingestionApplied at read time
Typical retention30 to 90 days hot1 to 7 years
Cost driverVolume ingested or indexedStorage, plus compute per query
Query latencySecondsSeconds to minutes
Built-in workflowAlerts, cases, dashboardsNone; you build or integrate it

In practice most organisations run both. High-value sources that feed live detections go to the SIEM at full fidelity; high-volume sources that are investigated rather than alerted on — flow records, DNS, proxy, firewall accepts, cloud audit trails — go to the lake. This split is also the most effective lever in SIEM cost optimisation, because the sources that dominate the licence bill are rarely the ones that produce alerts.

The Schema Problem, and OCSF

A lake full of raw logs in twenty vendor formats is a liability, because every query has to encode knowledge of each format. The Open Cybersecurity Schema Framework, announced in August 2022 by a group of vendors including AWS and Splunk, exists to fix this: it defines a shared event taxonomy so that an authentication event from any product lands in the same shape, with the same field names.

Amazon Security Lake, generally available since May 2023, builds directly on OCSF and Iceberg, and several other platforms now emit or accept the format. Whether or not you adopt OCSF specifically, adopt something. Normalising at write time costs engineering effort once; normalising inside every query costs it forever.

What You Gain

The Costs Teams Underestimate

A data lake moves work rather than removing it. Pipelines have to be built and maintained; partitioning and file sizes determine whether a query takes ten seconds or twenty minutes; compute charges arrive per query, so an unpartitioned full scan can cost more than a month of SIEM ingestion. Detection content written for the SIEM does not transfer, and there is no case management, alerting, or analyst interface unless you integrate one.

The honest prerequisite is a data engineering capability, or a managed service that supplies one. A small team without either will get more value from careful source selection and retention tiering inside their existing platform than from a lake nobody has time to maintain. A team already drowning in ingestion costs while deleting data they need will find the economics compelling.

Start by moving two or three high-volume, low-alert sources, prove that an analyst can query them during an investigation, and expand from there. At Privia Security, we help organisations design this kind of tiered monitoring architecture through our SOC maturation services.

FAQ – Frequently Asked Questions

Does a Security Data Lake Replace a SIEM?

Not for most organisations. A SIEM provides real-time correlation, alerting, case management, and an analyst workflow that a lake does not include. The common architecture keeps alert-generating sources in the SIEM with short retention and sends high-volume investigative sources to the lake with long retention, with federated search across both. Full replacement requires substantial in-house engineering.

What Is OCSF and Do We Need It?

The Open Cybersecurity Schema Framework is a vendor-neutral event taxonomy announced in 2022, giving security events consistent field names and structures across products. You do not have to use it, but you do need some normalisation standard. Without one, every analyst query must account for each source format, and the lake becomes unusable as sources are added.

How Much Data Should We Keep?

Work backwards from the questions you need to answer. Twelve months covers most retrospective hunting and regulatory enquiries. Longer periods, often three to seven years, are driven by sector requirements, litigation holds, or contractual commitments. Authentication, network flow, and cloud audit records justify the longest retention; verbose debug-level application logs rarely do.

Is a Data Lake Cheaper Than a SIEM?

Per gigabyte stored, by a wide margin. Overall, only if you account for the engineering time to build and run the pipelines, the compute charged per query, and the detection content that has to be rewritten. The saving is real for organisations with large volumes of investigative data; it is often illusory for those ingesting modest volumes without a data engineering function.

Related Terms

You May Be Interested In These

What Is Spyware?

Cybersecurity Terms

13 September 2021

What Is Spyware?

Read More
What Is Zigbee Security?

Cybersecurity Terms

17 November 2026

What Is Zigbee Security?

Read More
What Is Sysmon?

Cybersecurity Terms

17 November 2026

What Is Sysmon?

Read More