Load Testing

Baseline / ramp / soak / spike / stress test methodology, tools (k6, Locust, JMeter), and production traffic replay.

Overview

Why load testing exists

Load testing is how you validate that capacity planning is real, not aspirational. Without it, capacity numbers are guesses.

There are five test types, and each one answers a different question.

> [!IMPORTANT] The baseline is the reference. Every other test is compared against it.

The five test types

1. Baseline — Normal load — 15-30 min — p50/p95/p99 latency, CPU/memory/network/disk profile, error rate under normal conditions
2. Ramp — Baseline up to 5-10x — 30-60 min — The knee where latency climbs, the ceiling where errors appear, the bottleneck
3. Soak — Sustained, above baseline — Hours to days — Memory leaks, pool exhaustion, disk fill, cache pollution, thread or goroutine accumulation
4. Spike — Baseline to 5-10x in seconds, hold 5-15 min, drop — Minutes — Autoscaling reaction time, cold starts, pool resize speed, cold-cache surges
5. Stress — 2-5x the projected limit — Until failure — Whether the failure mode is graceful or cascading

Ramp: finding the knee

The ramp output is a statement such as "this service starts degrading at 5000 QPS and fails at 7000 QPS." Compare that to the forecast peak.

Soak and spike: time-dependent bugs

Memory leaks invisible in 30-minute tests — Degradation during the autoscaling ramp-up window
Connection-pool exhaustion that takes hours — Cold-start performance of new instances
Disk fill from slow-growing logs or temp files — Whether DB connection pools scale in time
Cache pollution that changes performance over time — Cold caches causing a downstream surge
Long-running goroutine or thread accumulation

> [!TIP] Soak tests are often run weekly or before a major release. They catch defects that unit and integration tests miss.

Stress: graceful vs cascading

> [!IMPORTANT] The point is not to pass the stress test. It is to verify that the failure mode is graceful.

Tools

k6 (Grafana Labs) — JavaScript — Modern CLI plus cloud; industry favorite for HTTP/gRPC; easy CI integration
Locust — Python — Distributed swarm workers; good for dynamic, scripted load; slower per worker than k6
JMeter — Java GUI — Older and heavyweight; fits enterprise environments that prefer Java
Gatling — Scala — Very high performance; best for hyperscale tests
Vegeta — Go — Simple, fast HTTP tester; less programmable than k6
wrk / wrk2 — C — Highest performance per worker; least feature-rich

Production mirror and shadow traffic

The most realistic test replays real production traffic at a staging service.

AWS VPC Traffic Mirroring for HTTP — Real header and cookie patterns
Kafka MirrorMaker for event streams — Real key distribution: hot users, hot products, fat tails
Custom shadow services routing prod traffic to staging — Real time-of-day variability and edge cases such as malformed requests and deprecated paths

> [!WARNING] Shadow traffic costs significant infrastructure, because the shadow service must keep up with prod. Reserve it for services where load-test fidelity matters most.

The discipline

Test regularly — Weekly soak tests, monthly ramp tests, pre-event stress tests; not only pre-launch
Prod-like environment — Same instance types, data sizes and dependencies; staging with 100 rows of data is not a load test
Test the failure mode — The system will hit stress levels in production; find the failure mode in a controlled setting first
Automate — Load tests in CI for every release, compared against the last release's baseline

Key concepts

Baseline / ramp / soak / spike / stress: Five test types. Each finds different bugs. Run all of them periodically.
k6: Modern JavaScript-based load tester. CLI + cloud. Industry favorite for HTTP/gRPC. Easy CI integration.
Knee of the curve: Load level where latency starts climbing nonlinearly. Capacity planning aims to keep peak below the knee.
Soak test bugs: Memory leaks, connection-pool exhaustion, disk-fill, slow accumulation issues. Invisible in 30-min tests.
Production traffic replay: Replay real prod traffic at staging. Highest-fidelity load test. Catches real-world distribution issues.
Stress test goal: NOT "pass" — verify failure mode is GRACEFUL. Cascade vs graceful is the distinction.

When to use it

Pre-launch — verify capacity claims
Pre-event — Black Friday, product launch
Post-architecture-change — did the new design hold up?
Quarterly — regression tests; new bottlenecks emerging

Interview questions

Walk me through the load test types and what each finds.
How do you set up production-realistic load tests?
I ran a stress test and the service crashed. Is that a failure?