Security Platform for a Hybrid, Multi-Cloud Fleet

Identity, network, data, workloads, detection and response without breaking five nines

Design identity, keys, guardrails and detection for 400 microservices across data centers, AWS, Azure and Google Cloud, so that no security service's outage breaks 99.999%.

Overview

A company runs 400 microservices. Some live in its two data centers, the rest across two regions each of AWS, Azure and Google Cloud, and the most important ones promise 99.999% availability. You are asked to design the security platform for all of it: how people and services prove who they are, how secrets and keys are kept, how traffic is segmented, how rules are enforced, how attacks are detected and how incidents are handled.

The trap in this question is that security and availability pull against each other. Every control you add is a new dependency: an identity provider that must answer before an engineer can fix an outage, a key service that must answer before a database can read its own disk, a policy webhook that must answer before a Pod can start. A design that calls those services on every request inherits their failures. Five nines leaves about 26 seconds of failure a month; a single five-minute key-service outage on the request path spends almost twelve months of it.

So the design has two halves. The first is zero trust across four estates: no request is trusted because of where it comes from; it carries an identity that is checked, authorized and logged. The second is making that identity layer survive its own failures: credentials are short-lived but cached and renewed early, decisions are made locally, each security service runs per site, and for each dependency you decide in advance whether its outage should block (fail closed) or let traffic through (fail open), and how you will notice.

Scope it first

How many services and where? — 400 services, about 4,800 instances, on 8 sites: 2 data centers and 2 regions in each of AWS, Azure and Google Cloud
What availability do the critical services promise? — 99.999% for the critical tier, 99.9% for internal tools
One identity provider for people today? — No: assume an on-premises directory plus each cloud's own; we consolidate on one IdP
A service mesh already? — Kubernetes everywhere new, a mesh on most clusters; some VMs on premises without one
Regulated data? — Payment card and personal data in a few services, so encryption, audit and evidence matter
Who responds to incidents? — One security operations team for all estates, on call around the clock
What is out of scope? — App-level security (passwords, OAuth, OWASP) and the build pipeline, covered by other topics

The interviewer listens for four things. Do you put identity, not network location, at the center, for people and for workloads? Do you keep a separate identity system per cloud or bridge them with federation? Do you treat each security service as an availability dependency and say what happens when it is down? And can you walk an incident from an alert in one cloud to containment in another?

Key concepts

Zero trust
Single IdP and SSO
Just-in-time access
Break-glass accounts
SPIFFE/SPIRE
Workload identity federation
mTLS in the mesh
Envelope encryption and key availability
Org guardrails as code
Fail-open vs fail-closed

Functional requirements

One sign-in for every engineer to every estate, with MFA and no standing administrator rights
Just-in-time elevation to privileged roles with approval, a time limit and an audit record
Break-glass accounts that work when the IdP or the network to it is down, and alert every time they are used
A cryptographic identity for every workload instance, issued automatically and rotated without restarts
Mutual TLS and identity-based authorization on every service-to-service call, across sites
Workloads reach cloud APIs with federated, short-lived credentials, never stored keys
Secrets and encryption keys managed centrally, rotated on a schedule, with access logged
Guardrails that block dangerous changes at the organization, account and cluster level
Continuous posture checks, runtime threat detection and one place to search security logs
An incident response process that can contain a compromised identity or workload in any estate

Non-functional requirements

No security service on the request path of a critical service without a local cache or local decision (our design rule)
Any central security service may be down for 30 minutes without user-visible failures (our target)
Credential lifetimes of an hour or less for workloads and of hours for people (our choice)
Revocation of a compromised identity takes effect everywhere within 15 minutes (our target)
Every access decision attributable to an identity and kept for at least a year (our retention choice)
Detection of a critical finding to a human looking at it within 15 minutes (our target)

Interview questions

The IdP goes down. What happens to the fleet?
How do services in the data center authenticate to Google Cloud and Azure without stored keys?
Fail open or fail closed for an OPA or Kyverno admission webhook?
How do you keep the key service from breaking availability?
How do you rotate secrets and certificates for 4,800 instances without outages?
How would you enforce the same guardrails in AWS, Azure and Google Cloud?
Walk me through an incident that crosses clouds.
Why one SPIFFE trust domain per site instead of one for the fleet?