Online Coding Judge

Run and grade untrusted code

Design a coding practice and contest site that compiles and runs code from strangers against hidden tests, safely and fairly, and returns a verdict within seconds even when a contest starts.

Overview

An online judge takes a program someone typed into a browser, compiles it, runs it against tests the author cannot see, and answers: Accepted, Wrong Answer, Time Limit Exceeded, and so on. Practice sites, programming contests and hiring assessments all work this way.

Two things make it a real design problem. First, the code is untrusted: it runs on your machines, and some of it will try to read the hidden tests, open network connections, fill the disk or fork until the machine falls over. Second, the load is bursty: when a contest starts, thousands of people submit within minutes, and every one of them is watching the clock.

So the design has a quiet web side (problems, submissions, a scoreboard) and a hard execution side (a queue, a fleet of workers, and a sandbox around every run).

Questions to ask first

How much traffic? — 1 million submissions a day; contests peak at 500 a second
Which languages? — C++, Java, Python, Go and a few more, each with its own time multiplier
How many tests per problem? — About 50 hidden tests, up to a few MB each
How fast must verdicts come? — Under 10 seconds at p95 outside contests (a target we chose)
Partial scores? — No: a submission is accepted only if every test passes
Can users run their own input? — Yes, a 'Run' button with a smaller limit

What you will be able to do

Accept a submission asynchronously and hand it to workers through a queue that survives crashes.
Build a sandbox from namespaces, cgroup v2 limits, seccomp and rlimits, and say when to add gVisor or a microVM.
Turn a run's measurements into the right verdict, including the sleeping program and the out-of-memory kill.
Size the worker fleet for a contest and keep contest submissions ahead of practice ones.
Keep a live contest scoreboard.

Key concepts

Judge queue
Worker leases
Namespaces
cgroup v2 limits
seccomp
gVisor
CPU vs wall time
Verdicts
Output checker
Warm pools

Functional requirements

Submit code for a problem in a supported language and get a verdict
Show per-submission results: verdict, first failing test number, time and memory used
Run code against the user's own input without submitting
Enforce time, memory, output and process limits per problem and language
Run contests with a live scoreboard
Rejudge all submissions of a problem after its tests are fixed

Non-functional requirements

Submitted code can never read tests, other users' code, or reach the network
A verdict within 10 seconds at p95 outside contests; contests stay under 30 seconds at peak (targets we chose)
The same code gets the same verdict on every run, within a small timing margin
A worker crash never loses a submission or produces two verdicts
Contest submissions are never stuck behind practice traffic

Interview questions

How do you run untrusted code safely?
How do you handle 10,000 submissions in the first minutes of a contest?
Why is the submission API asynchronous?
How do you decide between TLE, MLE and RE when a program is killed?
How do you make timings fair across languages and machines?
What happens if a worker dies in the middle of judging?
How would you add plagiarism detection?