Job Scheduler

Distributed cron and delayed jobs

Design a service that runs jobs at a set time or on a cron schedule across a fleet of workers, never loses a run, and keeps duplicate runs harmless.

Overview

A job scheduler runs work later: send this reminder in 24 hours, build the sales report every day at 06:00 New York time, clean up expired sessions every 10 minutes. On one machine, cron does it. Across a company it becomes a service: many teams submit jobs, a fleet of workers runs them, and nothing may be lost when any machine dies.

The interview is about three questions. How does the scheduler find the jobs that are due, among a hundred million, without scanning them all? How is each run given to exactly one worker at a time, and what happens when that worker dies halfway? And what does 'every day at 02:30' mean on the night the clocks change?

The worker side (leases, heartbeats, retries, dead-letter queues) is covered in depth in the topic Managing Long-Running Tasks. This problem focuses on the scheduling side and shows the claim in runnable PostgreSQL.

Questions to ask first

How many jobs, how many runs? — 100 million jobs; about 86 million runs a day
How precise must firing be? — Within a few seconds; not a real-time timer
What if a run is missed during an outage? — Run it late, or skip it, as each job chooses
Can a run happen twice? — Rarely, after a crash; jobs must be idempotent
Time zones? — Each recurring job has one; schedules follow local time
How long do runs take? — Seconds to an hour; long ones heartbeat

What you will be able to do

Store jobs so the due ones are found with one index scan, and turn them into runs safely from several schedulers.
Claim runs across many workers with FOR UPDATE SKIP LOCKED and a lease, and recover runs from dead workers.
Explain why the honest guarantee is at-least-once plus idempotency, and where duplicates come from.
Compute the next fire time of a cron expression in a time zone, including daylight saving changes.
Handle the top-of-the-hour spike and the backlog after an outage.

Key concepts

next_fire_at index
Due-job polling
FOR UPDATE SKIP LOCKED
Leases and reapers
At-least-once plus idempotency
One run per fire time
Cron and time zones
Timing wheels
Jitter and backlog draining

Functional requirements

Create a one-time job for a given time, or a recurring job from a cron expression and time zone
Run each due job on one of many workers, with a timeout
Retry failed runs with backoff, up to a limit
Pause, resume and cancel jobs
Show each job's runs: when, where, how long, and the result

Non-functional requirements

A due run starts within 5 seconds of its time in normal operation (a target we chose)
No run is lost when schedulers or workers crash
Duplicate runs are rare and harmless: jobs receive a run id to deduplicate with
Schedulers and workers scale out and have no single point of failure
After an outage, the backlog drains at a controlled rate

Interview questions

Can you guarantee a job runs exactly once?
How do many workers take runs without taking the same one?
How does the scheduler find due jobs among 100 million?
What happens to a job scheduled at 02:30 on the night clocks change?
What is a timing wheel and when would you use one?
How do you handle job dependencies (run B after A)?
How do you avoid the top-of-the-hour stampede?