CI/CD and GitOps for a Multi-Cloud and On-Premises Fleet

Build once, sign, promote by waves, never break more than one cell

Design the build, release and deployment system for hundreds of services on Kubernetes on-premises and on AWS, Azure and Google Cloud, so a bad change never takes down more than one cell.

Overview

A company runs about 400 services. Each one deploys a few times a day to Kubernetes clusters in its own data centers and on AWS, Azure and Google Cloud. The business promises 99.999% availability, which leaves about 5.3 minutes of full outage a year, or about 26 seconds a month. Many outages start with a change, so the delivery system is the main thing standing between a typo and the SLA.

The production fleet is split into cells: independent copies of the stack, each serving a fixed slice of users, with no shared runtime state between them (the Microservices at 99.999% on AWS, Azure and Google Cloud topic designs the cells themselves). The cell is the unit of blast radius. This design's job is to make sure a change reaches cells one at a time at first, that a bad change is caught and undone while it has hurt at most one cell, and that nothing the company did not build itself can ever run.

The design rests on three ideas. Build once: every change becomes one image, identified by its content digest, and that same digest is promoted from staging to the last cell; environments differ only in configuration. Trust what you can verify: the build platform signs a provenance statement saying what it built from which commit, and every cluster checks that signature before it runs a container. Pull, do not push: an agent in each cluster pulls its desired state from Git or a registry, so CI never holds cluster credentials and a cluster keeps converging even when the central systems are down.

Scope it first

How many services and how often do they deploy? — 400 services, about 3 releases each a day: 1,200 releases a day
Where do they run? — On-premises (3 data centers) and AWS, Azure, Google Cloud (3 regions each), 3 cells per region: 36 production cells
Is each cell a separate Kubernetes cluster? — Yes; a cell may hold several clusters, but one is enough here
What counts as a change? — Code, container base images, Kubernetes config, feature flags and infrastructure code
Who may change production? — Only the pipeline; people approve, the pipeline applies
What must still work during a cloud outage? — Building, signing and shipping a fix to the healthy cells and the recovering ones
Any regulated workloads? — Assume on-premises cells hold data that may not leave them, so their builds and registries must work without the cloud

The interviewer listens for four things. Do you promote the same artifact by digest instead of rebuilding per environment? Do you verify provenance at admission, not only sign it? Do you order the rollout by blast radius (one cell, then one region, then one cloud, then the rest) with bake times, and treat config and flags the same way? And can the delivery system ship a fix during an outage of a cloud, the registry or its own control plane?

Key concepts

Build once, promote the digest
SLSA provenance
Keyless signing (Sigstore)
Admission verification
Registry replication
GitOps pull model
ApplicationSets and fleet labels
Waves of cells
Bake time and canary analysis
OIDC workload identity federation

Functional requirements

Build each merged change once into an image identified by its sha256 digest, with tests, an SBOM and signed provenance
Copy the image and its attestations to a registry in every region of every platform before any cell there is asked to run it
Promote a release through ordered waves of cells, each gated on health, SLO analysis and a bake time
Roll back automatically when a gate fails, and roll the whole fleet back to the last good release in one action
Refuse at admission any image that is not signed by the company's build platform for an approved source
Treat config, feature flags and infrastructure changes as releases with their own waves
Support deploy freezes and an audited emergency path that still verifies what it ships

Non-functional requirements

Blast radius: a bad change reaches at most one cell before it is stopped (our target), and never two cells at once
Lead time: a normal release reaches all 36 cells in about 4 to 5 hours; an emergency fix in under an hour (our targets)
Availability of delivery: shipping a fix must work with any one cloud, the primary registry or the central hub down
No long-lived credentials in pipelines: every cloud and cluster credential is short-lived and scoped to one job
Auditable: every change in every cell traces to a commit, a build, an approver and a signature
Running cells never depend on the delivery system: if it stops, services keep running on their last good state

Interview questions

Why build once and promote the digest, instead of building per environment?
Walk me through how an image is signed and how a cluster decides to run it.
What do SLSA build levels mean, and which do you need?
How do you make sure a bad change takes down at most one cell?
Would you run one Argo CD for the whole fleet?
The registry is down. What breaks and what keeps working?
How does the pipeline get credentials to three clouds without storing keys?
A cloud is down and you must ship a fix in 30 minutes. What happens?