Managing Long-Running Tasks

How to run work that takes seconds to hours without holding a request open: accept and return a job id, queue the work, survive worker crashes with leases and idempotency, report progress, and allow cancellation.

Overview

Some work takes too long for a request and response: rendering a large report, transcoding a video, importing a million rows, training a model, sending a campaign to every customer. Such work can take seconds, minutes or hours, and the user should not wait on an open connection while it runs.

The standard answer is to split the request from the work. The API checks the request, records a job, puts it on a queue, and answers at once with the job's id. Background workers take jobs from the queue and do them. The client learns the result later by polling a status URL, through a push, or by a webhook.

That split is the easy part. The design work is in what happens when things go wrong: a worker crashes halfway, a job fails every time it runs, the queue grows faster than workers can drain it, a user cancels, or the same job runs twice. Interviewers listen for leases, idempotency, retries with a limit, and a clear state for every job.

What you will be able to do

  • Turn a slow endpoint into an asynchronous job with 202 Accepted and a status URL.
  • Model a job as a state machine and store it so any worker can pick it up.
  • Recover from crashed workers with leases and heartbeats, and make repeated runs harmless.
  • Handle failing jobs with backoff, a retry limit and a dead-letter queue.
  • Report progress, allow cancellation, checkpoint very long jobs and size the worker pool.

Key concepts

  • 202 Accepted and job ids
  • Job state machine
  • Job queues
  • Leases and heartbeats
  • At-least-once and idempotency
  • Retries and dead-letter queues
  • Progress and cancellation
  • Checkpoints
  • Worker capacity
  • Scheduled jobs

Interview questions

  • Design an API that generates large reports taking several minutes.
  • How do you detect and recover from a worker that dies in the middle of a job?
  • What does at-least-once delivery mean for your workers, and how do you cope with it?
  • How do you handle a job that keeps failing?
  • How would you let users see progress and cancel a running job?
  • How many workers do you need, and how do you scale them?
  • How do you run a job that takes 20 hours?
  • How do you make sure a nightly job runs exactly once?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio