Delivery Semantics

At-most-once, at-least-once and exactly-once: what each promise means, why a sender cannot tell a lost message from a lost reply, and how Kafka and idempotent consumers turn at-least-once delivery into an exactly-once effect.

Overview

A message system moves a message from a producer, through a broker such as Kafka or SQS, to a consumer that acts on it. Machines crash and networks drop packets, so every system has to say what happens to a message when something fails. That promise is its delivery semantics.

There are three levels. At-most-once: a message may be lost, but it is never handled twice. At-least-once: a message is never lost, but it may be handled more than once. Exactly-once: every message takes effect once, no more and no less. The first two come almost for free from the order in which you send, retry and acknowledge. The third has to be built.

The root cause is simple. When a sender gets no reply, it cannot tell whether its message was lost or only the reply was. If it retries, it may create a duplicate; if it does not, it may lose the message. Exactly-once therefore never means the bytes cross the network once. It means duplicates are detected and dropped, so the effect happens once.

What you will be able to do

  • Define the three guarantees and say which failure causes loss and which causes duplicates.
  • Explain why the consumer's offset commit order decides between at-most-once and at-least-once.
  • Build an idempotent consumer with a unique key, in the same transaction as the business write.
  • Explain how Kafka's idempotent producer, transactions and read_committed give exactly-once inside Kafka, and what it takes to extend that to an outside database.
  • Pick a guarantee for a given use case and say what it costs.

Key concepts

  • At-most-once
  • At-least-once
  • Exactly-once effect
  • Lost reply ambiguity
  • Offset commit order
  • Idempotent consumer
  • Idempotent producer
  • Kafka transactions
  • read_committed

Interview questions

  • What are at-most-once, at-least-once and exactly-once, and when do you use each?
  • Is exactly-once delivery possible?
  • How does Kafka achieve exactly-once semantics?
  • How do you build an idempotent consumer?
  • Why does the order of processing and offset commit matter?
  • Your payment consumer charged a customer twice. What happened and how do you fix it?
  • Does an SQS FIFO queue give you exactly-once processing?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio