Observability Across On-Premises and Three Clouds

Metrics, logs, traces and profiles that still work when a cloud is down

Design one observability platform for N microservices on-premises and on AWS, Azure and Google Cloud: metrics, logs, traces and profiles, SLOs and alerting, that still works when a region or a whole cloud is down.

Overview

A company runs about 200 microservices in four places: two on-premises data centers and two regions each in AWS, Azure and Google Cloud. A request from a user may start in AWS, call a service in Azure and read a database on-premises. When checkout gets slow at 3 a.m., the on-call engineer needs one place to ask 'what changed, where, and for which users?', and a page that arrives before the customers complain.

That is the job of the observability platform. It collects four kinds of telemetry: metrics (numbers over time, such as requests per second), logs (records of single events), traces (one request followed through every service it touches) and profiles (which lines of code use the CPU and memory). It turns them into SLOs (service level objectives, a target such as '99.9% of checkouts succeed over 30 days') and alerts, and it must stay useful when one region or one whole cloud is gone, which is exactly when people need it most.

The design is shaped by three forces. Every cloud has its own monitoring service and none of them sees the others. Telemetry is large (here about 9 TB a day before compression), and every cloud charges for data that leaves it. And a monitoring system that runs inside the thing it watches dies with it, silently.

Scope it first

How many services and where? — 200 microservices, 4,000 pods, on 8 sites: 2 data centers and 2 regions per cloud
How much traffic? — 20,000 requests a second on average, 50,000 at peak, about 8 spans per request
Which signals? — Metrics, logs and traces for every service; profiles for the busiest 20
Any existing tools? — Each cloud's native monitoring is on; the teams want one view, without lock-in
How long to keep data? — Metrics 13 months (downsampled), logs 14 days hot and 1 year cold, traces 14 days
What must page? — Breaches of user-facing SLOs, within about 5 minutes, even if a cloud is down
Compliance limits? — Logs with personal data stay in the region where they were written

The interviewer listens for four things. Do you standardize collection on one open format (OpenTelemetry) instead of four agents? Do you keep raw data near its source and still answer one question across all sites? Do you turn SLOs into burn-rate alerts that fire in minutes and stay quiet otherwise? And do you notice that a monitoring system can fail silently, and design how you would know?

Key concepts

OpenTelemetry SDK and Collector
OTLP
Semantic conventions
W3C Trace Context
Agent and gateway Collectors
Tail sampling
Local-first storage with federated queries
Cardinality and retention tiers
Multi-window, multi-burn-rate alerts
Dead man's switch and meta-monitoring

Functional requirements

Collect metrics, logs and traces from every service on all 8 sites in one format, and profiles from the busiest services
Find every span of one request by its trace ID, even when the request crossed clouds
Answer one query (a dashboard, a PromQL expression, a log search) across all sites, and say which sites did not answer
Define SLOs per user journey and evaluate them across all sites
Page the right on-call team within minutes of an SLO breach, with a runbook link and the evidence attached
Probe every public endpoint from outside the cloud that hosts it
Alert when the monitoring system itself stops working

Non-functional requirements

Survives the loss of any one site or any one cloud: alerting keeps working and says which data is missing (a target we chose)
Pages within about 5 minutes of a fast SLO burn (our target)
Telemetry is buffered through a network partition of at least 1 hour and delivered afterwards
Cross-cloud egress for telemetry stays a small fraction of raw volume (here under 0.1%)
Per-team limits on series and log volume, so one team cannot slow ingestion for all
No vendor lock-in at the instrumentation layer: services emit OTLP, and backends can change without code changes

Interview questions

Why not send everything to one central observability backend?
How does one trace survive crossing from AWS to Azure to on-premises?
Why does tail sampling need two Collector tiers?
Where does burn rate 14.4 come from?
One cloud is completely down. Why might the SLO alert not fire?
How do you know the monitoring system itself is working?
What happens to telemetry during a one-hour partition?
Would you use each cloud's native monitoring?