API Gateway

Unified API Entry Point

Design the gateway that every external API request passes through: it routes each request to a backend, checks the caller and their limits, and protects the services behind it, adding only a few milliseconds.

Overview

An API gateway is the single front door for a company's public API. A mobile app, a web page or a partner's server sends a request to api.example.com; the gateway decides which of about 200 internal services should answer it, checks the caller's credentials and limits, forwards the request, and passes the answer back.

This page is the interview walk-through: sizing the gateway, the request path, the data structures and the failure handling. What a gateway is for, when to add one, and how it differs from a load balancer or a service mesh are covered in the API Gateway Design topic; here we build one.

The interviewer is listening for three things. First, separating the fast data plane (the instances that handle requests) from the control plane (where routes and keys are managed), so a control-plane outage never stops traffic. Second, keeping the per-request work local and cheap: route matching in memory, token checks without a network call. Third, containing failures: a slow backend, a bad config push, or a lost zone must not become a full outage.

Questions to ask first

Who calls the API? — Our apps, and partners with API keys
How much traffic? — 100,000 requests a second on average, 300,000 at peak
How many services and routes? — About 200 services, about 2,000 routes
How do callers prove who they are? — Users: a JWT from our identity service. Partners: an API key
Are limits per caller? — Yes, per consumer and plan, and per route to protect backends
How often do routes change? — Many times a day, by many teams, with no restarts
Latency budget for the gateway? — Under 10 ms added at p99 (a target we chose)

What you will be able to do

Size a gateway fleet so it survives losing a zone at peak.
Separate the data plane from the control plane and push config safely.
Match 2,000 routes per request in microseconds with a path trie.
Verify JWTs and API keys without a network call on the hot path.
Stop retries and slow backends from turning into an outage.

Key concepts

Data plane and control plane
Route trie
Versioned config snapshots
JWT verification with cached keys
Hashed API keys
Distributed rate limits
Timeouts and retry budgets
Outlier ejection
Zone-sized capacity

Functional requirements

Route each request to a backend service by host, path prefix or pattern, and method
Authenticate callers by JWT or API key, and pass their identity to the service
Enforce rate limits per consumer and per route; answer 429 with Retry-After
Apply per-route timeouts and retries, and stop sending traffic to failing backends
Let teams add and change routes through an admin API, live, with rollback
Emit an access log line and metrics for every request

Non-functional requirements

Adds under 10 ms at p99 on top of the backend's time (a target we chose)
99.99% availability for the data plane (a target we chose)
Survives the loss of one zone at peak traffic with no extra capacity added
Keeps serving on the last good config if the control plane is down
A bad config change reaches a small canary before everyone

Interview questions

How does the gateway change routes without restarts or downtime?
How many gateway instances do you need?
How do you rate limit across many gateway instances?
Where do you validate tokens, and how without calling the auth service?
A backend is timing out. What does the gateway do?
How do you stop the gateway from being a single point of failure?
What should never go into the gateway?