CI/CD Pipeline

Hosted build and test service

Design a hosted service that runs a repository's build, test and deploy jobs on every push, in isolated machines, and streams their logs live.

Overview

A CI/CD service runs the checks a team wants on every change. A developer pushes a commit; the service reads a pipeline file from the repository, splits it into jobs (install, lint, test, build, deploy), runs each job on a machine called a runner, and reports pass or fail back on the commit. Continuous integration is the build-and-test part; continuous delivery or deployment adds the jobs that ship the result.

It is a scheduling problem with a security problem inside it. Jobs have an order, so the service must run them as a graph and start each one as soon as what it needs has finished. Demand comes in bursts, so the pool of runners must grow and shrink. And a job runs code the service did not write, often from strangers who opened a pull request, so every job gets a clean machine and only trusted runs see secrets.

Questions to ask first

Who are the users, and how many runs? — A hosted service for many organizations; 2 million pipeline runs a day
What starts a run? — Pushes, pull requests, tags, schedules and manual starts
How are jobs defined? — A YAML file in the repository; each job lists the jobs it needs
Do we run untrusted code? — Yes: pull requests from forks of public repositories
Do we deploy too? — Yes, as ordinary jobs; the rollout strategy is a separate system
How fresh must logs be? — Visible within a couple of seconds while the job runs

What you will be able to do

Turn a pipeline file into a job graph and schedule it on the critical path.
Design a job queue that survives runner crashes, using leases.
Size and autoscale a runner pool from the traffic numbers.
Decide which runs may see secrets and write caches, and why.
Stream live logs without putting them through the database.

Key concepts

Webhook triggers
Job graph (DAG)
Critical path
Job queue with leases
Ephemeral runners
Autoscaling
Content-keyed caches
Artifacts
Secret masking
Concurrency groups

Functional requirements

Start a run on push, pull request, tag, schedule or a manual request
Read the pipeline file at that commit and build a job graph from it
Run each job on a fresh, isolated runner once the jobs it needs succeed
Restore and save dependency caches; upload and download artifacts
Give secrets to trusted runs only, and mask them in logs
Stream each job's log live, and keep it after the job ends
Cancel a run; let only one deploy per environment run at a time

Non-functional requirements

A job starts within 30 seconds of becoming ready at p95 (a target we chose)
Log lines reach a watching browser within 2 seconds (a target we chose)
No job can read another job's files, memory or secrets
A runner crash never loses a job silently: it is retried or marked failed
The service keeps accepting triggers when runners are short; jobs wait in the queue

Interview questions

How do you turn a pipeline file into running jobs?
A runner dies halfway through a job. What happens?
How do you run code from strangers safely?
How do you make builds fast?
How do you stream logs to thousands of watching developers?
How do you size the runner pool?
How do you stop two deploys from racing?