Explore our new Cybersecurity Terms Glossary, explained by our experts.
Cybersecurity Terms Glossary
Investigating a Kubernetes compromise means reading the API server audit log, the cluster state, and node-level telemetry, because the workloads themselves are usually gone before anyone looks.

What Is Kubernetes Incident Analysis?
A Kubernetes cluster is a control loop that keeps replacing things. Pods are rescheduled, nodes are replaced by the autoscaler, and a compromised container may have been deleted and recreated a dozen times before an alert is triaged. Investigating an incident here means working from the records of what the cluster was asked to do, rather than from the artefacts of what it did.
Every action in a cluster passes through the API server, which can log each request with the authenticated identity, the source address, the verb, the resource, and the response. Configured at RequestResponse level for sensitive resources, the audit log answers the central questions: which identity created the privileged pod, when the role binding changed, who read the secrets, and from where.
This is also where preparation decides outcomes. Managed services do not all enable audit logging by default, or they retain it briefly, and self-managed clusters need an audit policy written by hand. A cluster without an audit log leaves an investigator reconstructing events from fragments, and the honest finding in that situation is that the question cannot be answered.
Kubernetes events are a weaker substitute. They are useful, but they live in etcd with a default retention of about an hour, so they are gone long before most investigations start unless they are shipped elsewhere.
The instinct to delete a suspicious pod destroys everything. A better sequence is to cordon the node so nothing new is scheduled onto it, apply a network policy that isolates the pod rather than terminating it, and set the workload’s replica count so the controller does not immediately recreate what you are examining. Then capture node memory, preserve the container’s writable layer as described in Docker container forensics, export the pod specification and the relevant cluster state, and snapshot any persistent volumes.
Runtime security tooling is what fills the gap for workloads that no longer exist. Falco, Tetragon, and equivalent eBPF-based sensors record process execution, file writes, and network connections per container and ship them off the node immediately, which converts an ephemeral workload into a durable record.
One compromised pod is rarely the whole story. Establish which service account the workload used and what that account could reach, whether secrets were mounted into it, whether the node hosted pods from other namespaces or tenants, and whether the cluster’s cloud identity grants access to resources outside it — a node role with broad permissions turns a container compromise into a cloud incident. Then rotate: service account tokens, secrets that were mounted, and any cloud credentials the workload could reach.
Audit logging at a sensible level with off-cluster retention, control plane and node logs shipped to a central store, runtime sensors in at least detect mode, admission control that blocks privileged pods, and image digests recorded at deployment. The NSA and CISA Kubernetes hardening guidance covers the preventative side of the same list. Our cloud security services include assessing whether a cluster could actually be investigated today, which is a different exercise from assessing whether it is hardened.
The API server audit log. Every action in the cluster passes through the API server, so the log records which identity created or modified each resource, when, and from which address. Without it, you are reconstructing events from node logs and container output. Check that it is enabled, set to a useful level, and retained off-cluster before you need it.
Not as a first step. Deleting it destroys the container’s writable layer, its process memory, and often the only record of what ran. Cordon the node, isolate the pod with a restrictive network policy, and prevent the controller from immediately recreating it. Then capture memory, preserve the filesystem layer, and export the cluster state before removing anything.
About an hour by default, because they are stored in etcd with a short time to live. They are useful during live triage and effectively useless afterwards unless they have been shipped to a central log store. Treat them as a supplement to the audit log rather than a substitute, and configure forwarding if you want them available later.
Not necessarily, but you have to prove it rather than assume it. Establish what the pod’s service account could access, whether secrets were mounted, whether the container was privileged or mounted host paths, and what cloud permissions the node identity holds. Any of those can turn a single workload compromise into cluster or cloud account compromise.
You May Be Interested In These
We use a few cookies that are needed to make this website work, and they are always on. We would also like to set analytics cookies, to see which pages are useful; functional cookies, so that our live chat works; and advertising cookies, to measure our campaigns. We will not set any of those until you say yes, and you can change your mind at any time from the Cookie Settings link in the footer.
Choose the cookies you are happy for us to use. Strictly necessary cookies are always on; everything else is off until you turn it on.
These cookies keep the website working: they hold your security checks, your form submissions and the choice you make here. The site cannot run without them, so they are always on.
These let our live chat open and remember your conversation, and let the newsletter sign-up work. Switch them off and the chat window will not appear.
These tell us which pages are read and how people find us, so that we can improve the site. The reports are about groups of visitors, never about you personally.
These let us measure our advertising and show you our adverts on other websites, such as LinkedIn, Google and Meta. Switch them off and you will still see adverts, but they will be less relevant.