Monitoring and Observability
How to see what a running system is doing: metrics, logs and traces, the signals worth measuring, SLOs and error budgets, and alerts that wake people only when users are hurt.
Overview
Monitoring means collecting numbers and events from a running system and raising an alarm when something is wrong. Observability is the wider goal: being able to answer new questions about the system from the data it gives out, without shipping new code to find out. When a request crosses ten services, you cannot log in to one server and look around; the data the services send out is all you have.
That data comes in three kinds. Metrics are numbers counted over time, such as requests per second. Logs are records of single events. Traces follow one request through every service it touches. Each answers a different question, and a good setup links them so you can move from one to the next.
The other half of the topic is deciding what 'working' means. Google's SRE practice does this with service level objectives (SLOs): a target such as '99.9% of requests succeed this month'. The gap between the target and perfection is an error budget, and alerts fire when that budget is being spent too fast. That keeps pages rare and tied to what users feel.
What you will be able to do
- Say which of metrics, logs and traces answers a given question, and how to link them.
- Choose the metrics a service should expose, using the golden signals, RED and USE.
- Define an SLI and an SLO for a service, and work out its error budget.
- Design burn-rate alerts that page for real user harm and stay quiet otherwise.
- Explain liveness, readiness and startup probes, and the mistake that turns a probe into an outage.
Key concepts
- Metrics, logs and traces
- Symptoms vs causes
- Golden signals, RED and USE
- Percentiles and histograms
- Label cardinality
- Distributed tracing
- SLIs, SLOs and error budgets
- Burn-rate alerts
- Health probes
- Blameless postmortems
Interview questions
- What are metrics, logs and traces, and how do you use them together in an incident?
- Which metrics should every service expose?
- Define SLI, SLO and SLA, and show how an error budget works.
- Design the alerting for a service with a 99.9% availability SLO.
- How does distributed tracing work, and how do you choose a sampling strategy?
- Why do we use percentiles instead of averages, and how do you compute a p99 across many servers?
- What is the difference between liveness and readiness probes, and what goes wrong with them?
- Walk me through how you would handle a production incident.
Sign up free to read the full topic
More System Design topics
- Metrics Monitoring and Alerting
- Microservices
- ML Experiment Tracking System
- ML Feature Store
- ML Training Data Lake
- Model Registry and Artifact Store
- Monolith vs Microservices
- Multi-Modal AI Search Engine
- Multi-Modal Data Store
- Multi-Region Active-Active Architecture
- Multi-Step Processes
- Multi-Tenancy Architecture
Start free on Cariara · Coding problems · System design studio