Webhook Delivery Service

Reliable event delivery to customer endpoints

Design the service that sends a signed HTTPS POST to every customer endpoint subscribed to an event, retries through outages, and never lets one slow endpoint delay anyone else.

Overview

Your company's product produces events: an invoice was paid, an order shipped, a user signed up. Customers want to hear about them at once, without polling an API. So each customer registers one or more endpoints: an HTTPS URL on their own servers plus the event types they care about. When an event happens, the delivery service sends a signed HTTP POST with the event to every endpoint that subscribed to that type.

The service is easy to sketch and hard to run, because the receiving servers belong to customers. Some answer in 300 ms. Some take 10 seconds, return 500 for a day, or vanish. The requirement that shapes everything is isolation: one slow or failing endpoint must never delay delivery to anybody else. The second shaping fact is that delivery is at least once. When a request times out, the sender cannot know whether the receiver processed it, so it must send again, and receivers deduplicate on the event ID.

This problem is the full interview design. For the concepts on their own (push versus poll, payload styles, how a receiver should verify), see the Webhook System Design topic.

Questions to ask first

How many events, and how many endpoints per event? — 50 million events a day; 3 subscribed endpoints per event on average
How bursty is traffic? — Peak is 5 times the average
How big is a payload? — About 2 KB of JSON
How fast are endpoints? — A healthy one answers in about 300 ms; we stop waiting after 10 s
What guarantee do customers get? — At least once, with a stable event ID; no ordering unless an endpoint asks for it
How long do we keep history? — 30 days of delivery log, for the dashboard and replay
How long do we keep retrying? — Up to three days, then the delivery is marked failed and kept for replay

What you will be able to do

Turn 50 million events a day into deliveries a second, requests in flight and terabytes of log, and say which number drives the design.
Use Little's law to show why a shared worker pool fails and size a per-endpoint cap.
Design retries with exponential backoff and full jitter, a give-up rule, a dead-letter state and auto-disable.
Sign deliveries with HMAC-SHA256, reject replays, and rotate a secret without dropped events.
Explain why there is no ordering by default, what ordered delivery costs, and how to stop customers from pointing you at your own network.

Key concepts

Transactional outbox
Fan-out per endpoint
Per-endpoint in-flight cap
Little's law
Circuit breaker
Backoff with full jitter
HMAC-SHA256 signatures
At-least-once delivery
SSRF protection
Replay

Functional requirements

Customers register an HTTPS endpoint and choose event types; they can update, pause and delete it
Every event is delivered as a signed POST to every enabled endpoint subscribed to its type
Failed deliveries are retried with backoff for up to three days, then marked failed
Customers can rotate the signing secret with an overlap window
Customers can see every attempt (status, latency, response snippet) for 30 days and replay one delivery or a time range
An endpoint can opt in to ordered delivery

Non-functional requirements

No event is lost once the product's transaction commits (at-least-once delivery)
Isolation: a slow or failing endpoint never delays delivery to other endpoints
First attempt within 5 s of the event for p99 of healthy endpoints (a target we chose)
The management API is available 99.9% of the time; delivery degrades by delaying, never by dropping
Workers never send a request to a private, loopback or metadata address
Writes scale to 5 times the average rate without manual work

Interview questions

Why can't one slow endpoint be solved by adding more workers?
How does a dispatcher keep one endpoint from blocking a whole queue partition?
Walk me through what happens when a delivery times out.
What retry schedule would you use, and when do you give up?
How do receivers know a webhook really came from you?
A customer asks for events in order. What do you offer?
How do you stop customers from using your webhooks to reach your internal network?
How would the design change at ten times the load?