Kubernetes Security Events to Prioritize

Kubernetes generates an overwhelming volume of events, and most teams either alert on noise or miss the handful that signal an actual compromise. The events that matter are few and specific: someone exec-ing a shell into a running pod, a privileged or host-mounting container being created, and changes to cluster-admin RBAC. Those are the steps between a foothold and full cluster control. This guide ships the Falco rules and audit-log detections for the events worth your pager.

Kubernetes attacks map cleanly to MITRE ATT&CK’s container matrix: T1610 — Deploy Container, T1611 — Escape to Host, and T1078 — Valid Accounts for abused service-account tokens. The events below are where those techniques surface.

What makes a Kubernetes event worth alerting on?

Kubernetes is chatty — every reconcile loop, scale event, and health check is a log line. Alerting on volume guarantees fatigue. The signal comes from events that are rare in normal operations and necessary for an attack: getting a shell inside a container, deploying a pod configured to reach the host, or granting yourself cluster-admin. Those are not part of a healthy deploy pipeline.

It is the same rare-and-bad principle behind every darkpwn detection, applied to the cluster — the discipline from the detection engineering workflow carried into Kubernetes telemetry.

Which Kubernetes security events should you prioritize?

These map to the attacker’s path: deploy or access a container, configure it to reach the host, and escalate via RBAC. Cover this list before tuning anything else.

What does the Kubernetes audit log actually capture?

The audit log is configurable to a degree that catches people out, and the default configuration in most clusters cannot support half the detections written against it.

Every request is recorded at one of four levels, set per-resource by the audit policy:

This matters immediately. The KQL detection above keys on RequestObject containing "privileged":true — and RequestObject only exists at Request or RequestResponse level. On a cluster auditing pods at Metadata, that rule is syntactically valid, runs without error, and can never match. It is the single most common reason a Kubernetes detection appears deployed and is not.

Managed control planes complicate this further, because you do not own the audit policy:

• AKS exposes two diagnostic categories: kube-audit and kube-audit-admin. The second omits get and list events, which is where the overwhelming majority of volume lives. Sending kube-audit-admin to the SIEM and the full kube-audit to cheap storage is usually the correct split — the admin stream carries every event in the priority table above.
• EKS offers control-plane log types including audit and authenticator, written to CloudWatch. The audit policy itself is fixed by AWS, so verify what level pods are captured at rather than assuming.
• GKE routes equivalent data through Cloud Audit Logs, with admin activity and data access split into separate streams that are billed and retained differently.

Volume is the other planning constraint. get, list, and watch from controllers and operators dominate audit output by a wide margin and carry almost no detection value. Filtering them at the source rather than at the SIEM is what makes Kubernetes audit logging affordable, and it is the difference between a cluster whose logs you keep for a year and one whose logs you keep for a week.

Which RBAC permissions are actually escalation paths?

RBAC reviews tend to focus on obviously alarming grants like cluster-admin. The permissions that matter most in practice are the ones that look narrow and are not.

The two rows worth internalising are create pods and nodes/proxy. Anyone who can create a pod in a namespace can run as any service account in that namespace, which means pod-creation rights are transitively equal to the most privileged service account present. And nodes/proxy grants kubelet access, which provides container execution on everything scheduled to that node without any pods/exec grant appearing in the role.

Also worth a standing alert: any binding to the system:masters group. Membership in that group bypasses RBAC authorisation entirely — the authorizer short-circuits before evaluating any role. There is essentially no legitimate reason for a new binding to appear.

How to detect the events that matter

Two sources cover it: Falco for runtime (syscall-level) behavior, and the Kubernetes API audit log for control-plane actions.

A shell spawning inside a container is the runtime tell — the same service-process-spawns-a-shell logic that catches web RCE, applied to pods.

- rule: Shell Spawned In Container
  desc: A shell was executed inside a running container
  condition: >
    spawned_process and container
    and shell_procs and not known_shell_entrypoints
  output: >
    Shell in container (user=%user.name container=%container.name
    image=%container.image.repository proc=%proc.cmdline)
  priority: WARNING
  tags: [container, mitre_execution, T1610]

On the control plane, the audit log shows escape-enabling pod specs. In KQL (Sentinel / AKS diagnostics):

KubeAuditLog
| where Verb == "create" and ObjectRef.resource == "pods"
| where RequestObject contains "\"privileged\":true"
   or RequestObject contains "\"hostPID\":true"
   or RequestObject contains "\"hostPath\""
| project TimeGenerated, User.username, ObjectRef.namespace, SourceIPs

How to prevent the events you’re detecting

1. Enforce Pod Security Standards (restricted profile) via admission control to block privileged, host-mounting, and host-namespace pods by default.
2. Least-privilege RBAC — no wildcard verbs/resources, no anonymous bindings, and tightly held cluster-admin, per the NSA/CISA Kubernetes Hardening Guidance.
3. Limit exec to break-glass and audit every use.
4. Bind service-account tokens to their audience and rotate them; alert on off-cluster use.

How Pod Security Standards actually work

PodSecurityPolicy was removed in Kubernetes 1.25. Its replacement, Pod Security Admission, is a built-in admission controller configured with namespace labels rather than cluster-wide policy objects — which makes it far simpler to adopt and far easier to leave half-applied.

Three profiles, applied per namespace:

• privileged — no restrictions. This is the default for any namespace you have not labelled, which is the important half of the sentence.
• baseline — blocks the known escape paths: privileged containers, host namespaces, hostPath volumes, most capability additions.
• restricted — baseline plus hardening: runAsNonRoot, allowPrivilegeEscalation: false, a RuntimeDefault seccomp profile, and dropping ALL capabilities.

Three modes, and you can set all three at once:

The adoption path that works is warn and audit at restricted first, across every namespace, with enforce left unset. You then get a complete list of what would break, delivered to the people whose workloads would break, without breaking anything. Turn on enforce namespace by namespace as each one comes clean.

Two caveats worth knowing before you rely on it. Pod Security Admission evaluates the pod, so a Deployment with a non-compliant template is accepted while the pods it creates are rejected — the failure surfaces in the ReplicaSet’s events, not in the kubectl apply, and it confuses people the first time. And an unlabelled namespace is unrestricted, so a security control that depends on labels needs its own control ensuring the labels exist. Alerting on namespace creation without security labels closes that gap.

Service account tokens are no longer what you remember

Two changes reshaped this attack surface, and detections written before them are looking for the wrong thing.

Bound service account tokens are now the default: projected into the pod, audience-scoped, time-limited, and tied to the pod’s lifetime. A token lifted from a compromised pod expires and is rejected by any audience it was not issued for — a substantial improvement over the old model.

Automatic Secret generation stopped. Kubernetes no longer creates a long-lived token Secret for every service account. Legacy long-lived tokens still exist in older clusters and still work, and they are the ones actually worth hunting for: they never expire, are audience-unscoped, and are frequently copied into CI systems and configuration repositories.

Two detections follow directly. Alert on explicit creation of a service-account token Secret, which is now a deliberate act rather than a default. And alert on any service-account authentication originating outside the cluster’s own address ranges — the audit log carries the system:serviceaccount:<namespace>:<name> identity and the source IP, and a workload identity appearing from outside the cluster means the token left the pod.

What does Falco actually cost to run?

Falco is the right recommendation and it is not free, and guides that present it as a drop-in miss the decisions that determine whether it survives its first quarter.

• It needs a driver on every node. The kernel module, the eBPF probe, or modern eBPF with CO-RE. Modern eBPF avoids compiling anything per-kernel, which is the operational headache that causes most Falco deployments to rot after an OS upgrade. Check your kernel supports it before choosing.
• Syscall monitoring costs CPU on every node. The overhead is usually modest and it is not zero, and it scales with syscall volume — meaning your busiest, most latency-sensitive nodes pay the most.
• Managed control planes are out of reach. On EKS, AKS, and GKE you cannot run a driver on control-plane nodes. Falco covers your workloads; the API server’s behaviour comes from the audit log only. Any threat model that assumes runtime visibility across the whole cluster is wrong on a managed platform.
• The default rules are a starting point, not a deployment. Out of the box they will fire on legitimate operator and CI behaviour. Budget real tuning time, or the alerts become background noise within a fortnight.

If that is more than you can operate, the honest fallback is genuinely useful: enforce Pod Security Standards through admission control, and detect from the audit log alone. You lose in-container runtime visibility, which is a real loss, and you keep every detection in the priority table that comes from the control plane. That is a defensible position for a small team, and it is a much better outcome than a Falco deployment nobody maintains.

How do you triage a Kubernetes security alert?

Cluster alerts have a distinctive property: the identity that performed the action is usually a service account belonging to a pipeline, not a person, so “who did this” needs one more hop than it does on an endpoint.

1. Resolve the identity to a human or a system. The audit log gives you system:serviceaccount:<namespace>:<name> or a user. A service account means the next question is which workload holds that token, and whether the action is something that workload has ever legitimately done.
2. Check the source IP against the cluster’s ranges. An in-cluster address means a workload made the call; an external address means either a human with a kubeconfig or a token that has left the cluster. These are very different incidents.
3. Establish whether this is normal for that principal. CI service accounts create pods constantly. The question is never “did a pod get created” but “did this principal create this kind of pod,” and answering it needs a baseline you built before the alert.
4. Pull the full request body. For a pod-creation alert, the spec tells you the image, the service account, the volumes, and the security context — everything you need to judge intent. If the body is missing, fix the audit level; you will need it again.
5. Look at what the pod did after it started. Falco events, network egress, and API calls made using its service account. A privileged pod that ran nothing is a misconfiguration; one that mounted the host filesystem and read /etc/shadow is an intrusion.
6. Check the image and its provenance. An unknown registry, a :latest tag from a public source, or an image not present in your registry is a strong signal by itself, and connects to supply chain attack detection.
7. Enumerate what that service account can reach, not just what it did. Its RBAC bindings and any cloud IAM role bound to it define the actual blast radius, and cloud role bindings are the part that turns a container compromise into an account compromise.
8. Preserve before you delete. Deleting a pod destroys its container filesystem and memory. Capture what you need first — the instinct to remove the offending workload immediately is the Kubernetes equivalent of killing a process before taking a memory image.

Step 7 is where cluster incidents most often turn out to be bigger than they looked. A workload identity federated to a cloud IAM role means the container escape you are investigating may already be a cloud incident, and that determination changes who needs to be in the room. Follow it into CloudTrail monitoring patterns.

Common Kubernetes detection mistakes

• Alerting on volume. Reconcile noise drowns the real events.
• Audit log only, no runtime. You miss in-container behavior; add Falco.
• No admission control. You detect privileged pods you could have blocked.
• Ignoring RBAC drift. A wildcard binding is a quiet path to cluster-admin.
• Writing spec-inspecting rules against Metadata-level auditing. The rule is valid and can never fire. Verify RequestObject exists before trusting anything that reads a pod spec.
• Labelling some namespaces for Pod Security Admission. An unlabelled namespace is unrestricted, so the control needs a control: alert on namespaces created without security labels.
• Reviewing RBAC for cluster-admin only. create pods, pods/exec, and nodes/proxy are the grants that actually appear in real escalation paths, and none of them look alarming in a role definition.
• Assuming runtime coverage on a managed cluster. You cannot run a Falco driver on a managed control plane. The API server is audit-log-only, whatever your architecture diagram shows.
• Deleting the pod before capturing it. Container filesystems and memory do not survive termination, and neither does your ability to say what happened.
• Ignoring the cloud IAM role bound to a service account. It is frequently the most privileged thing a compromised pod can reach, and it is invisible in Kubernetes RBAC.

Kubernetes security events checklist

1. Ship both the API audit log and Falco runtime events to the SIEM.
2. Alert on exec into pods and on privileged/host-mounting/host-namespace pod creation.
3. Alert on anonymous/wildcard RBAC bindings and cluster-admin changes.
4. Alert on service-account tokens used off-cluster.
5. Enforce Pod Security Standards (restricted) via admission control.
6. Apply least-privilege RBAC per NSA/CISA hardening guidance.
7. Correlate audit-log intent with Falco runtime behavior by pod.
8. Validate each detection by generating the benign event in a test cluster.

The takeaway

Kubernetes security monitoring is a short, high-signal list: exec into pods, escape-enabling pod specs, and cluster-admin RBAC changes, detected by correlating Falco runtime events with the API audit log and backed by Pod Security Standards. Continue with CloudTrail monitoring patterns and command injection detection, then follow the attacker sideways through the service mesh with zero trust microservices east-west detection and check how the workload got there in supply chain attack detection. Or browse the full Detection Engineering pillar.