Zero Trust Microservices: East-West Detection

Zero trust microservices architecture is widely deployed and narrowly understood. Most estates that describe themselves as zero trust have enabled mutual TLS between services and stopped there, which authenticates every caller and authorizes none of them. The result is a flat network with excellent encryption: any compromised service holds a valid identity certificate and can reach every other service in the mesh. This guide covers detecting east-west movement in that environment, and closing the authorization gap that makes it possible.

East-west movement inside a mesh maps to MITRE ATT&CK T1021 — Remote Services, T1550.001 — Use Alternate Authentication Material: Application Access Token, and T1078.004 — Valid Accounts: Cloud Accounts. It is the cloud-native counterpart to the Windows techniques in detecting lateral movement, and it has one significant defensive advantage those techniques do not.

What is zero trust for microservices?

Zero trust for microservices means network position confers no trust. A service does not get to call another service because it happens to sit inside the same cluster, VPC, or namespace. It gets to make that call because it presents a cryptographic identity and because an explicit policy permits that identity to invoke that method on that service.

Three properties are required, and most deployments implement only the first:

The gap between rows two and three is where nearly all real risk lives. Deploying a mesh and turning on strict mTLS is a well-documented afternoon of work with a visible, satisfying result: all traffic encrypted, all callers authenticated. Writing authorization policy for every service pair is weeks of unglamorous work requiring knowledge of which services legitimately call which, and it is frequently deferred indefinitely.

Why is a microservice call graph easier to baseline than a user network?

This is the defensive advantage worth exploiting, and it is why east-west detection in a mesh can be dramatically higher fidelity than lateral movement detection on a corporate network.

A user network’s connection graph is driven by human behaviour, which is irregular, seasonal, and legitimately novel all the time. People open new applications, visit new file shares, and start new projects. Any “never seen this connection before” rule on a user network drowns in true positives that are not incidents.

A microservice call graph is driven by code. Service A calls service B because an engineer wrote a client call in a source file. The edge set changes when someone deploys, and it does not otherwise change at all. In a mature system with a hundred services, the set of legitimate directed edges is typically a few hundred pairs, it is enumerable, and it is stable for weeks.

That stability is what makes the primary detection viable:

A service pair that has never communicated in thirty days and suddenly communicates is either an undocumented deploy or an intrusion. There is no third common explanation.

In a system I operate — roughly ninety-five services across four languages behind a gRPC mesh — the observed edge set stays constant between deploys to a degree that makes new-edge alerting practical rather than aspirational. The operationally important detail, learned the hard way, is that the rule must be correlated with deployment events. Without that correlation every release produces a burst of alerts for genuinely new edges and the rule gets muted within two sprints.

What telemetry do you need?

The sixth row is the most commonly missed and it is a genuine blind spot rather than a tuning problem. Teams instrument their mesh carefully and then move a large share of inter-service communication onto Kafka or a similar bus, which does not traverse the mesh at all. Every mesh authorization policy and every mesh detection is silently bypassed for that traffic. If your architecture uses an event bus, its authorization and telemetry need equal attention, and it usually gets none.

How to detect east-west movement

Three analytics covering the mesh, the identity layer, and the message bus.

Service-pair call never seen in the baseline topology

index=mesh sourcetype=envoy:access earliest=-1h
| eval caller = coalesce(source_principal, downstream_peer_id)
| eval callee = coalesce(destination_principal, upstream_cluster)
| eval edge = caller . " -> " . callee . " :: " . request_method
| search response_code!=0
| lookup mesh_edge_baseline edge OUTPUT first_seen, call_count_30d
| where isnull(first_seen)
| join type=left caller
    [ search index=cicd sourcetype=deploy:complete earliest=-6h
      | eval caller = "spiffe://" . cluster . "/ns/" . namespace . "/sa/" . service_account
      | fields caller, deploy_id, deploy_time, release_version ]
| eval recent_deploy = if(isnotnull(deploy_id), "YES", "NO")
| stats count AS calls, values(source_workload) AS caller_pods,
        values(destination_workload) AS callee_pods,
        values(request_method) AS methods,
        values(response_code) AS codes,
        max(recent_deploy) AS after_deploy
        by edge, caller, callee
| where after_deploy = "NO"
| sort - calls

The deployment join is the difference between a rule that runs for years and one that gets muted in a fortnight. A new edge within six hours of a deploy of the calling service is a release, and suppressing it removes essentially all of the benign volume. What remains is a new call path with no corresponding code change, which is precisely the anomaly worth waking someone for.

Rebuild mesh_edge_baseline on a rolling thirty-day window, and treat a growing baseline as useful architectural documentation in its own right.

Workload identity used from an unexpected pod or namespace

A stolen service account token or projected certificate is used from the attacker’s location rather than from the workload it was issued to. The identity is valid; its origin is not.

let baseline_window = 30d;
let detect_window = 1h;
let identity_baseline =
    KubeAuditLogs
    | where TimeGenerated between (ago(baseline_window) .. ago(detect_window))
    | where isnotempty(ServiceAccount)
    | summarize KnownPods = make_set(SourcePod, 200),
                KnownNamespaces = make_set(SourceNamespace, 50),
                KnownNodes = make_set(SourceNode, 100),
                KnownVerbs = make_set(Verb, 50)
            by ServiceAccount;
KubeAuditLogs
| where TimeGenerated > ago(detect_window)
| where isnotempty(ServiceAccount)
| join kind=inner identity_baseline on ServiceAccount
| extend NewPod = SourcePod !in (KnownPods),
         NewNamespace = SourceNamespace !in (KnownNamespaces),
         NewVerb = Verb !in (KnownVerbs)
| where NewNamespace or (NewPod and NewVerb)
| extend RiskScore = toint(NewNamespace) * 4
                   + toint(NewPod) * 2
                   + toint(NewVerb) * 2
                   + iff(Verb in ("create","delete","patch","escalate","impersonate"), 3, 0)
| where RiskScore >= 5
| project TimeGenerated, ServiceAccount, SourceNamespace, SourcePod, SourceNode,
          Verb, Resource, NewNamespace, NewPod, NewVerb, RiskScore
| order by RiskScore desc

A namespace change weighs heaviest because it is the strongest indicator that the token left the workload it belongs to. Pod change alone is normal, since pods are rescheduled constantly, which is why it only scores when combined with a verb the identity has never used. The escalate and impersonate verbs are worth a separate, immediate alert regardless of score, as they connect directly to the container-escape paths in Kubernetes security events to prioritize.

Message bus topic consumed by an unauthorized identity

The blind spot. Bus traffic bypasses the mesh, and topic authorization is frequently permissive because it was configured during development and never tightened.

index=kafka sourcetype=kafka:authorizer earliest=-24h
| eval principal = replace(principal_name, "^User:CN=([^,]+).*", "\1")
| lookup topic_acl_baseline topic principal
     OUTPUT authorized, expected_operation, consumer_group_expected
| eval finding = case(
      isnull(authorized),                              "UNDECLARED_TOPIC_ACCESS",
      operation != expected_operation,                 "OPERATION_MISMATCH",
      operation == "Read" AND consumer_group != consumer_group_expected,
                                                       "UNEXPECTED_CONSUMER_GROUP",
      operation == "Describe" AND topic == "*",        "TOPIC_ENUMERATION",
      1==1,                                            null())
| where isnotnull(finding)
| stats count AS events, values(topic) AS topics,
        values(operation) AS operations,
        values(client_host) AS hosts,
        values(consumer_group) AS groups,
        min(_time) AS first_seen
        by principal, finding
| eval topic_count = mvcount(topics)
| sort - topic_count, - events

TOPIC_ENUMERATION is the reconnaissance signal and deserves priority. A legitimate service knows which topics it needs, because they are in its configuration. A service enumerating all topics is either a misconfigured admin tool or something mapping your event architecture, and the topic_count sort surfaces broad access patterns first.

Which false positives will you actually see?

The batch-job row is a subtle one worth designing around. A job that runs monthly will look like a new edge on a thirty-day baseline every single time it runs. Set the baseline window longer than your longest legitimate cycle, or explicitly register scheduled jobs as known edges.

How do you triage suspected east-west movement?

1. Identify the compromised workload, not just the anomalous call. The new edge tells you the source identity; the actual question is what happened inside that pod.
2. Pull the full call history for that identity. Everything it called in the last 24 hours, whether allowed or denied. Denials map the attacker’s exploration.
3. Check whether the identity was used from more than one pod. This separates a compromised workload from a stolen credential, and the responses differ.
4. Enumerate what the identity could reach. Its authorization policy and its RBAC bindings. Capability, not observed use.
5. Check the API server audit log for the service account. Token theft frequently precedes direct API server access.
6. Check the message bus separately. Mesh telemetry does not cover it, and it is a common second stage.
7. Isolate rather than delete the pod. Cordon and snapshot before terminating, since a deleted pod takes its evidence with it. This is the cloud-native equivalent of pulling the network cable rather than powering off.
8. Rotate the workload identity and any secrets it could read, following the secrets rotation path.
9. Determine initial access. A vulnerable dependency, an exposed endpoint, or a supply chain compromise reaching the build.

Step 7 is where cloud-native response most often goes wrong. The instinct under pressure is to kill the pod, and killing the pod destroys memory, process state, and local filesystem evidence irretrievably while the orchestrator cheerfully schedules a replacement.

How to test your east-west detection

In a non-production cluster you own:

1. Deploy two services with no legitimate call path between them and have one call the other. Confirm the new-edge rule fires and the deploy-join does not wrongly suppress it.
2. Deploy a genuinely new service with a new call path. Confirm this is suppressed, verifying the suppression works in the intended direction.
3. Copy a projected service account token into a pod in a different namespace and use it. Confirm the workload identity rule scores it high on the namespace change.
4. Have a service consume a Kafka topic outside its ACL. Confirm the bus rule fires, and note whether any mesh rule fired, which demonstrates the bypass.
5. Enumerate topics with a non-admin principal. Confirm TOPIC_ENUMERATION fires.
6. Confirm mesh access logs actually carry the caller principal. This is the single dependency that silently invalidates everything above, and it is frequently absent by default.

Item 6 should be step one in practice. Many mesh configurations omit the source principal from access logs by default, and every analytic here evaluates to nothing without it.

How to build actual zero trust between services

Four further controls carry disproportionate weight:

• Never mount broad service account tokens. Disable automounting by default and grant narrowly. Most workloads never need API server access at all.
• Segment the message bus by topic and principal. Read on one topic is not read on all topics.
• Run workloads as non-root with read-only root filesystems. This raises the cost of turning code execution into a durable foothold.
• Treat the mesh control plane as tier-zero. Whoever controls policy distribution controls every authorization decision in the estate.

Which zero trust control should you deploy first?

The ordering is deliberate. You cannot write default-deny policy without knowing your real call graph, and attempting it without that knowledge produces an outage followed by a permissive rollback. Baseline first with detection, use the baseline to author policy, then enforce.

Common zero trust microservices mistakes

• Treating mTLS as the finished state. It authenticates; it does not authorize.
• Permissive default authorization. Every service reachable from every service.
• Omitting the caller principal from access logs. This makes east-west detection impossible.
• Ignoring the message bus. It bypasses every mesh control you deployed.
• Automounting service account tokens everywhere. Free API server credentials in every pod.
• New-edge alerting without deploy correlation. The rule gets muted after one release.
• Deleting the pod during response. Evidence gone, replacement scheduled.
• Baseline windows shorter than your job cycles. Monthly batch jobs alert monthly.

Zero trust microservices checklist

1. Confirm mesh access logs include the caller principal, not only the callee.
2. Capture the invoked method and the authorization decision on every call.
3. Export mesh authorization denials as a high-priority alert stream.
4. Build a rolling thirty-day service-pair call-graph baseline.
5. Alert on service-pair edges absent from the baseline, correlated with deploy events.
6. Extend the baseline window beyond your longest scheduled job cycle.
7. Collect API server audit logs and baseline service account usage by pod and namespace.
8. Alert on workload identity used from an unexpected namespace.
9. Alert immediately on the escalate and impersonate verbs regardless of score.
10. Apply and monitor message bus ACLs per topic and principal.
11. Alert on topic enumeration by non-operator principals.
12. Author default-deny authorization policy per service pair using the baseline.
13. Move to short-lived automatically rotated workload certificates.
14. Disable service account token automounting by default.
15. Run workloads non-root with read-only root filesystems.
16. Rehearse the isolate-before-delete response so evidence survives.

The takeaway

Zero trust microservices means authorization, not just encryption. mTLS with permissive policy is a well-encrypted flat network in which any compromised workload can reach everything. Exploit the one advantage cloud-native gives you: a call graph defined by code is stable enough that a new service-pair edge is genuinely anomalous, so baseline it, correlate with deploys, and alert. Extend the same thinking to workload identity origin and to the message bus that bypasses your mesh entirely. Then use that baseline to author the default-deny policy that makes the movement impossible rather than merely visible. Continue with Kubernetes security events to prioritize, CloudTrail monitoring patterns that matter, and detecting lateral movement, or browse the full Defensive Research pillar.