End-to-End Correctness and the Exactly-Once Myth

Why transactions and exactly-once messaging inside one component do not stop duplicate effects across the whole system, and how request IDs checked at the end points close the gap.

Overview

The problem: retries happen at every layer

A user presses Pay. The request crosses a browser, a network, a load balancer, an API server, a database and maybe a message broker. Any of them can time out, and the usual fix for a timeout is to try again. The trouble is that a timeout does not tell you whether the first attempt worked. It may have succeeded, and only the reply was lost.

Nothing here is a bug in a single component. The database gave full ACID guarantees both times. The failure is that nobody could tell the second request was the same as the first.

The end-to-end argument

In 1981 Saltzer, Reed and Clark presented a design principle at the Second International Conference on Distributed Computing Systems; a revised version appeared in ACM Transactions on Computer Systems in 1984: some functions can only be done completely and correctly with the knowledge of the application at the end points. Lower layers can help, but only as a performance improvement. One of their examples is duplicate suppression: even if the network removes duplicate messages, the application can create duplicates through its own retries, and those look like new messages to the network. So the application needs its own duplicate check, and once it has one, that check also covers duplicates from below.

TCP — Retransmitted segments within one connection — A retry on a new connection
Message broker — Duplicate writes from one producer session — A client resubmitting the same order
Database transaction — Nothing: it runs whatever it is given, atomically — That two transactions mean the same payment
Application with a request ID — Any repeat of the same logical operation — Nothing it was designed for

What exactly-once really covers

Kafka offers exactly-once processing for one specific path: read from Kafka topics, process, and write to Kafka topics. It works by committing the output records and the consumer's offsets in one Kafka transaction. The guarantee stops at Kafka's edge. If the consumer also charges a card or writes to a database, those side effects are not part of the Kafka transaction.

For outside systems there are two sound patterns. Either store the consumer's position in the same place, and the same transaction, as the output, or make the effect idempotent so that repeating it changes nothing. The common result is sometimes called effectively once: messages may arrive more than once, but the effect happens once.

Idempotency keys: one ID per operation

The client creates a unique ID for each logical operation, for example a random UUID generated when the user opens the checkout, and sends the same ID with every retry. The server records the ID in the same database transaction as the effect, behind a unique constraint. A repeat finds the ID already there and returns the saved result instead of acting again.

ON CONFLICT DO NOTHING returns no row for a key that already exists, so the second insert into charges has nothing to insert. The unique constraint does the real work: even two requests racing at the same moment cannot both claim the key.

Calling the outside world

The card network is not inside your database transaction, so the pattern has three steps. First, record the intent and claim the key. Next, call the provider and pass your own idempotency key along, so the provider also suppresses repeats. Finally, save the provider's answer against the key. If the server crashes between the claim and the save, a retry finds the key with no saved answer; it should repeat the provider call with the same key (which is safe, because the provider deduplicates it) rather than treat the request as done.

Stripe's API is a public example of this contract. Clients send an Idempotency-Key header on POST requests; Stripe saves the status code and body of the first request for a key, whether it succeeded or failed (including 500 errors), and returns the same result for repeats. It compares the parameters of a repeat with the original and returns an error if they differ. Keys can be up to 255 characters and may be removed once they are at least 24 hours old, after which the same key starts a new request.

Who creates the key — The client, once per logical operation — Only the client knows two sends are the same intent
Key format — Random UUID (v4) or similar — Timestamps and counters collide across clients
Same key, different body — Reject — It is a client bug, not a retry
Where to store it — Same transaction as the effect — Otherwise a crash can save one without the other
How long to keep it — Longer than any client retries — A repeat after expiry becomes a new operation

Consumers: store the offset with the effect

A message consumer faces the same problem. If it applies a change and crashes before recording its position, it will reprocess the message after restart. Either record the position in the same database transaction as the change, or give each message an ID and record processed IDs behind a unique constraint.

Checking correctness end to end

The same argument applies to data integrity. Disks, memory and software can corrupt data without reporting an error, so each layer's checks cover only that layer. The TCP checksum, for example, is a 16-bit ones' complement sum: good at catching random bit errors on the wire, but weak and blind to anything that happens before sending or after receiving. Strong end-to-end checks are cheap insurance.

Application-level checksum over the whole object — Corruption anywhere between writer and reader
Rebuilding a derived view from the log and comparing — Bugs in the code that maintains the view
Periodic reconciliation (ledger vs provider statements) — Lost or duplicated effects at system boundaries
Append-only records with request IDs — Makes every effect traceable to one request

What you will be able to do

Explain why ACID transactions and broker exactly-once do not prevent a double charge.
State the end-to-end argument and apply it to duplicate suppression.
Design an idempotency-key table with a unique constraint, in the same transaction as the effect.
Extend idempotency to an outside provider and to message consumers.
Name the end-to-end checks that catch what per-layer checks miss.

Key concepts

End-to-end argument: The principle, from Saltzer, Reed and Clark, that some guarantees can only be completed at the application end points; lower layers can help only as optimizations.
Idempotency key: A unique ID the client attaches to one logical operation and repeats on every retry, so the server can apply the operation once.
Idempotent operation: An operation whose repeat has no further effect, such as setting a value or inserting by a unique key. Adding to a balance is not idempotent.
Effectively once: At-least-once delivery combined with idempotent processing, so each message may arrive more than once but takes effect once.
Kafka exactly-once: Transactions that commit output records and consumer offsets together, for read-process-write paths that stay inside Kafka.
Deduplication table: A table of seen request or message IDs behind a unique constraint, written in the same transaction as the effect.
Reconciliation: Comparing your records with an independent record, such as a provider statement, to find lost or duplicated effects.

When to use it

Designing a payment, order or booking API that clients can safely retry.
Explaining why Kafka exactly-once does not stop duplicate database writes or charges.
Designing a consumer that updates a database from a stream without double-applying messages.
Discussing how to verify data integrity across a pipeline of systems.

Interview questions

How do you make a payment API safe to retry?
Kafka supports exactly-once. Why can a consumer still apply a message twice?
What is the end-to-end argument, and how does it apply to data systems?