Reliability, Scalability, and Maintainability

The three qualities every data system is judged on: correct answers when parts fail, steady performance as load grows, and a system people can run and change for years.

Overview

Every data system is judged on three qualities. Reliability: it keeps giving correct answers when hardware breaks, software has bugs and people make mistakes. Scalability: when load grows, there is a clear way to add resources and keep performance steady. Maintainability: many engineers can run it, understand it and change it for years.

These words are easy to say and hard to make precise. A strong interview answer turns each one into something you can measure and a technique that improves it, and says what that technique costs.

Reliability: faults are normal, failures are not

A fault is one part not doing its job: a dead disk, a crashed process, a bad config value. A failure is the whole system no longer giving users the service it promised. Faults cannot be prevented, so reliable systems are built to tolerate them: they detect a fault and contain it before it becomes a failure.

Faults come in three kinds, and they need different defenses.

Hardware — Disk, memory or power supply dies — Mostly random and independent — Redundancy: replicas, RAID, spare machines
Software — A bug that crashes every node on the same bad input — Correlated: hits many machines at once — Testing, isolation, gradual rollouts, watchdogs
Human — A wrong config value pushed to production — Common and often slow to fix — Safe tools, staged rollouts, quick rollback, good monitoring

Hardware faults are rare per machine but constant at scale. Disk makers quote a mean time to failure of 1,000,000 to 1,500,000 hours, which implies under 1% of disks fail each year. A field study of over 100,000 drives found yearly replacement rates above 1% as a rule, commonly 2 to 4%, and up to 13% on some systems. At 3% a year, a cluster of 10,000 disks replaces about 300 disks a year, or almost one a day. So a large system is always running with something broken.

Human error deserves the most attention. A study of three large Internet services found operator error was the single largest cause of failures in two of them, and configuration mistakes were the most common kind of operator error.

Scalability: describe the load, then measure performance

The question 'does it scale?' has no answer on its own. First say which load is growing, using load parameters: the few numbers that describe the load on this particular system. Then say how you measure performance, and what you will do as the numbers grow.

Social feed — Posts per second, timeline reads per second, followers per account
Payments — Transactions per second at peak, share that are refunds
Chat — Concurrent connections, messages per second, group size
Analytics — Rows ingested per day, bytes scanned per query

Twitter's home timeline is the classic example. In a 2012 talk, Twitter's slides gave about 5,000 new tweets a second on an average day and more than 300,000 home timeline queries a second. Reads outnumbered writes by roughly 60 to 1, so Twitter did extra work when a tweet was posted (copying it into followers' timelines) to make reads cheap. The Fan-Out topic covers that design.

There are two ways to measure performance. Throughput is how much work gets done per second. Response time is how long one request takes, from the client's point of view. Report response time as percentiles (p50, p95, p99), never only an average, because the average hides the slow requests users complain about. The SLAs, SLOs and Percentile Latency topic goes deeper.

Scaling up means a bigger machine: simple, but with a ceiling and a single point of failure. Scaling out means more machines: no hard ceiling, but data must be split or copied. Stateless services (they keep no data between requests) scale out easily, since any copy can serve any request. Stateful ones, like databases, need replication or partitioning, and those bring cross-partition queries, rebalancing and consistency problems.

Maintainability: most of the cost comes after launch

Most of the cost of software is paid after the first version ships: fixing bugs, running it, adapting it to new needs. Maintainability has three parts.

Operability — Is it easy to run day to day? — Good metrics and logs, safe deploys and rollbacks, predictable behavior, runbooks
Simplicity — Can a new engineer understand it? — Clear abstractions, fewer moving parts, removing accidental complexity
Evolvability — Is it easy to change? — Loose coupling, schema evolution, tests, small reversible changes

A helpful split is essential complexity, which comes from the problem the users actually have, versus accidental complexity, which comes from how we chose to build it (tools, performance workarounds, historical leftovers). You cannot remove the first. Good abstractions remove much of the second.

The three pull against each other

More replicas — Reliability, read scalability — Consistency, cost, operational load
Partitioning (sharding) — Write scalability — Simplicity: cross-shard queries and transactions
Adding a cache — Scalability, latency — Correctness: stale reads; one more system to run
Fewer, simpler components — Maintainability — Headroom for scale

What you will be able to do

Tell a fault from a failure, and name a defense for hardware, software and human faults.
Describe a system's load with its own load parameters before saying how it scales.
Measure performance with throughput and response-time percentiles.
Split maintainability into operability, simplicity and evolvability, with one practice for each.
Say what each reliability or scaling choice costs in the other two qualities.

Key concepts

Reliability: The system keeps working correctly, at the expected performance, even when faults happen.
Fault: One component not doing its job: a dead disk, a crashed process, a wrong config value.
Failure: The system as a whole stops giving users the required service. Fault tolerance keeps faults from becoming failures.
Load parameters: The few numbers that describe load on this particular system, such as reads per second, writes per second or followers per account.
Throughput: How much work the system completes per unit of time, such as requests or records per second.
Response time percentile: The time within which a given share of requests finish. The p99 is the time 99% of requests beat; 1 in 100 is slower.
Scale up and scale out: Scaling up moves to a bigger machine. Scaling out adds more machines and spreads the work across them.
Operability, simplicity, evolvability: The three parts of maintainability: easy to run, easy to understand, easy to change.
Accidental complexity: Complexity that comes from how the system was built rather than from the users’ problem. It can be reduced with better abstractions.

When to use it

Opening a system design answer: state the reliability target, the load parameters and how the system will be run.
Justifying replication, redundancy or safe-deploy practices by the faults they tolerate.
Answering how a design scales: name the load that grows and the bottleneck it hits first.
Arguing for or against a new component on maintainability grounds.

Interview questions

What is the difference between a fault and a failure, and why does it matter?
How do you define and measure scalability for a design?
Why is human error treated as a first-class reliability problem?
How would you make a service survive losing one machine without users noticing?
What makes a system maintainable in practice?