Message Queues

How a queue lets one service hand work to another: acknowledgments and timeouts, dead-letter queues, ordering, idempotent consumers, and how RabbitMQ, SQS, Kafka and Google Pub/Sub differ.

Overview

A message queue is a buffer between two programs. One program, the producer, puts a message on the queue and moves on. Another, the consumer, takes the message off later and does the work. Neither has to wait for the other, and neither has to be running at the same moment.

That one idea solves several problems. A web request can return in milliseconds while the slow part, such as sending an email or resizing a video, happens in the background. A burst of traffic piles up in the queue instead of knocking over the service behind it. And when a worker crashes, its message is not lost, because the queue gives it to another worker.

The price is a set of new questions that a direct call never raised. When is a message finished? What happens to one that fails every time? Can the same message arrive twice, and in what order do messages arrive? This chapter answers them with the documented behavior of four widely used systems: RabbitMQ, Amazon SQS, Apache Kafka and Google Cloud Pub/Sub.

What you will be able to do

  • Choose between a point-to-point queue and a publish/subscribe topic, and build fan-out on each product.
  • Explain what happens when a worker crashes mid-task: acknowledgments, visibility timeouts and dead-letter queues.
  • Keep related messages in order while still processing them in parallel.
  • Make consumers idempotent, so that a duplicate delivery does no harm.
  • Say when a log such as Kafka fits better than a queue, and compare RabbitMQ, SQS, Kafka and Pub/Sub.

Key concepts

  • Point-to-point
  • Publish/subscribe
  • Acknowledgments
  • Visibility timeout
  • Dead-letter queue
  • FIFO and message groups
  • Competing consumers
  • Idempotent consumer
  • Backpressure
  • Queue vs log

Interview questions

  • When would you put a queue between two services, and when would you not?
  • What is the difference between point-to-point and publish/subscribe, and how do you build fan-out?
  • A worker crashes halfway through a message. What happens?
  • How do you handle a message that fails every time?
  • How do you keep all the events for one order in sequence while still processing in parallel?
  • The queue delivers at least once. How do you make sure a customer is not charged twice?
  • The backlog keeps growing. What do you check, and what do you do?
  • Kafka or a queue such as SQS or RabbitMQ: how do you choose?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio