Self-healing

Kubernetes liveness probes, autoscaling, automated runbooks, and the limits of self-healing.

Overview

Recover without a human

Self-healing means building systems that recover from failures without human intervention. Fewer pages mean fewer mistakes and more availability per engineer-hour invested.

Six layers

1. Process (auto-restart) — Process crashes; supervisor restarts it — systemd, Docker restart policies, Kubernetes liveness probes — Memory leaks, deadlocks, panics, transient bugs — Doesn't fix the cause; crash loops enter back-off
2. Instance (auto-replace) — Instance fails health check; LB removes it, orchestrator replaces it — Kubernetes Deployment + readiness probe; AWS ASG health checks — Bad host, network issue, hardware failure — Doesn't help when all instances are overloaded
3. Capacity (autoscaling) — Load rises or falls; instances added or removed — Kubernetes HPA (CPU/memory), VPA, Karpenter (cluster), KEDA (event-driven), AWS ASG target tracking — Traffic spikes, slow ramps — Doesn't fix bugs or an overloaded dependency
4. Topology (auto-failover) — Region or AZ fails; traffic routes to a healthy one — Route 53 health checks, AWS Global Accelerator, Cloudflare Load Balancer, Cilium — AZ outages, partitions, regional failures — Doesn't fix data corruption or application bugs
5. Application (graceful degradation) — Dependency fails; serve cached, partial or fallback responses — Circuit breakers, fallback methods, feature flags — Slow or failing downstreams — Needs application work; not a free win
6. Runbook automation (auto-remediate) — Alert fires; automated runbook mitigates — PagerDuty Rundeck, StackStorm, custom Lambda + alert webhooks — Known patterns: "disk full → expand volume," "queue backed up → scale workers," "stuck process → restart" — Only patterns you've seen before

Kubernetes self-healing primitives

Liveness probe — "Is the process alive?" — Kill and restart
Readiness probe — "Is the process ready to serve?" — Remove from service endpoints
Startup probe — "Has the process finished initializing?" — Holds off the other probes for slow starters
PodDisruptionBudget — "Minimum healthy pods during voluntary disruption" — Prevents autoscaling from breaking the service
HPA — Scale pods on a metric (CPU, custom) — Adds or removes pods
Karpenter / Cluster Autoscaler — Scale nodes when pod requests exceed capacity — Adds nodes

Probes are critical and dangerous

Liveness tests "the process is functional," not "the request can be served" — A liveness probe that fails when the DB is slow kills all your pods: a cascading outage
Readiness can be more aggressive — Removing from endpoints is reversible; killing is not
Startup probes for slow starters — Don't kill a Java app that needs 3 minutes to warm up because liveness fails at second 30

> [!WARNING] Too aggressive, a liveness probe kills healthy pods (false-positive crash loops); too lenient, dead pods linger.

Limits of self-healing

Doesn't fix design errors — A service that crashes every 1000th request restarts forever, and nobody fixes the bug
Hides failure modes — Auto-remediation can mask a developing problem until it's catastrophic
Complicates debugging — "The pod restarted twice; the metric spiked; what was the cause?" Automation erased the evidence

> [!IMPORTANT] Every auto-remediation should alert humans even when it succeeds, so the underlying pattern gets reviewed. "Auto-restart succeeded" is information; investigate the cause.

Key concepts

Liveness vs readiness probe: Liveness = "kill if failing"; readiness = "remove from endpoints if failing." Liveness more dangerous; tune carefully.
Autoscaler: Adjust capacity based on signal. HPA (pod-level), VPA (resource-tuning), Karpenter (node-level), KEDA (event-driven).
Circuit breaker: Stop calling a failing dependency for X minutes; serve fallback. Prevents cascading failure.
Auto-remediation: Automated runbook fires on alert. "Disk full → expand"; "queue backed up → scale." Saves pages.
Probes can amplify outages: A probe that hits a database fails when DB is slow → kills all pods → outage. Probes should test process health, not full system.
Auto-remediation alerts: Even when auto-remediation succeeds, log/alert that it fired. Otherwise patterns are invisible until catastrophic.

When to use it

Designing a new service — set up probes, autoscaling, replication from day 1
Reviewing pages — which pages could be auto-remediated? Likely 50%+
Capacity planning — autoscaling parameters for traffic patterns
Postmortems — was a self-healing layer not in place that should have been?

Interview questions

How do you design Kubernetes probes correctly?
When does auto-remediation HELP and when does it HURT?
How do you design autoscaling policies?