A/B Testing Platform Design
How to build the system that runs controlled experiments: stable assignment, many experiments at once, exposure logging, a metrics pipeline, and the statistics that keep results honest.
Overview
An A/B test (a controlled experiment) shows two or more versions of something to randomly chosen groups of users and measures which one does better on a chosen metric. Version A is the control, usually the current product. Version B is the treatment, the change being tested. Because users are split at random, a difference in the metric that is too big to be chance can be credited to the change.
An A/B testing platform is the system that lets many teams do this safely at once. It decides which variant each user sees, records who actually saw what, joins that with what users did, and computes the results with the right statistics.
The engineering is not hard to sketch. The difficulty is that small mistakes quietly produce wrong answers: a user who flips between variants, a logging bug that drops users from one group, or a team that stops the test the first day it looks good. A strong interview answer designs the system and the safeguards together.
What you will be able to do
- Design assignment that gives each user the same variant on every server, with no stored state.
- Run many experiments at once with layers, and keep conflicting ones apart.
- Build the pipeline from exposure logs to a results scorecard.
- Work out how many users an experiment needs, and read a p-value and a confidence interval correctly.
- Explain why peeking inflates false positives, and how to detect a broken experiment.
Key concepts
- Randomization unit
- Stable hash assignment
- Layers and domains
- Holdout group
- Exposure logging
- Sample size and power
- p-value and confidence interval
- Peeking and sequential tests
- Sample ratio mismatch
- CUPED
Interview questions
- Design an A/B testing platform for a large consumer product.
- How do you make sure a user always sees the same variant?
- How do you run hundreds of experiments at once without them interfering?
- How many users does an experiment need?
- A product manager wants to stop the test on day two because it is significant. What do you say?
- Results show a big win, but treatment has 2% fewer users than control on a 50/50 split. What now?
- How would you make experiments reach a decision faster without more traffic?
- How do you know the platform itself is working?
Sign up free to read the full topic
More System Design topics
- ACID vs BASE
- Active-Active vs Active-Passive
- Ad Click Event Aggregation
- AI Agriculture Advisory System
- AI Airline Booking Chatbot
- AI Analytics Dashboard
- AI Anomaly Detection System
- AI Art Generation System
- AI Banking Chatbot
- AI Bot and Scalper Detection System
- AI Click Fraud Detection System
- AI Cold Start Recommendation System
Start free on Cariara · Coding problems · System design studio