Fault Tolerance in Stream Processing

How a stream processor survives crashes without losing or double-counting events: microbatches, checkpoints with barriers, transactional and idempotent outputs, and what exactly-once really promises.

Overview

A batch job that fails can simply run again: its input is fixed and its output is written only at the end. A stream job never ends. It holds state (counts, windows, join tables) and keeps sending results to other systems while it runs. When a machine crashes halfway, the processor must answer two questions: which events had already been counted, and which results had already gone out?

The goal is usually called exactly-once. It does not mean each event is processed only once; after a crash some events are processed again. It means the visible effect is as if each event had been processed exactly once: nothing lost, nothing counted twice. Some people say effectively-once for that reason.

At-most-once — Some events are lost — Acknowledge input before processing; no replay
At-least-once — Some events affect the output twice — Replay input from the last saved position
Exactly-once (effect) — Neither — Replay, plus atomic or idempotent state and output

Microbatching: reuse batch recovery

One approach cuts the stream into small batches and runs each like a tiny batch job. Spark Structured Streaming works this way by default. Every source must have offsets, like Kafka offsets, so the engine can re-read any range. Before each batch it records the offset range it is about to process in a checkpoint and write-ahead log. If a batch fails, the engine re-runs exactly that range. A sink that is idempotent absorbs the repeat, and Spark's guide says replayable sources plus idempotent sinks give end-to-end exactly-once. Not every sink qualifies: Spark's Kafka sink, for example, is documented as at-least-once, so readers of that topic must deduplicate.

The cost is latency: results appear once per batch, not per event.

Checkpoints with barriers: snapshot a running dataflow

Flink processes events one at a time and takes periodic consistent snapshots of the whole job instead. Its method is inspired by the Chandy-Lamport distributed snapshot algorithm and described in the paper on Asynchronous Barrier Snapshotting (Carbone and others). It is not the original algorithm: the default aligned mode saves no in-flight records, and Flink's docs say the unaligned mode below is the closer match.

The sources inject a barrier into the stream, tagged with the checkpoint number, and record their input offsets.
Barriers flow with the records and never overtake them, so each barrier splits the stream into records before the snapshot and records after it.
An operator with several inputs waits until the barrier has arrived on all of them (alignment), holding back records from inputs that are ahead. Then it saves its state to durable storage and passes the barrier on.
When every operator has saved its part, the checkpoint is complete.

On a failure, Flink restarts every operator from the latest completed checkpoint and rewinds the sources to the offsets saved in it. Events after that point are processed again, but the state they update was also rolled back, so internal state ends up counting each event once.

Aligned checkpoint (default) — Waits for barriers on all inputs — Exactly-once state; a slow input delays the snapshot
Unaligned checkpoint — Lets barriers overtake in-flight records and saves those records as part of the state — Faster checkpoints under backpressure; more data written per checkpoint
At-least-once mode — Skips alignment and keeps processing all inputs — Lower latency; events may be counted twice after recovery

Output is the hard part

Rolling back state is easy because the processor owns it. Output is not: once a result has gone to another system, a rollback cannot pull it back. Replaying events after a crash would then send the same result twice. There are three ways out.

Transactional sink — Write output inside a transaction that commits only when the checkpoint completes — Kafka topics; databases with transactions
Idempotent writes — Write so that repeating a write changes nothing: upsert by key, or keep the IDs of applied events — Key-value stores, databases
Store the position with the output — Save the input offset in the same atomic write as the result, and resume from it — A single database that holds both

For a Kafka sink, Flink uses a two-phase commit tied to checkpoints. Consumers of the output must set isolation.level=read_committed; otherwise they also read records from transactions that are later aborted.

The sink needs two settings. Each job needs its own transactionalIdPrefix so jobs on the same cluster do not interfere. And Flink's docs advise setting the Kafka producer's transaction.timeout.ms well above the longest checkpoint plus the longest restart; if Kafka expires a transaction before Flink commits it, that output is lost. Watch the numbers: Kafka's own producer default is 60 seconds, but Flink's KafkaSink builder sets 1 hour unless you override it, and brokers reject any timeout above transaction.max.timeout.ms, which defaults to 15 minutes. So either lower the sink's timeout to fit under the broker limit or raise that limit. Output also becomes visible only at checkpoint time, so the checkpoint interval adds to end-to-end latency.

Exactly-once inside Kafka Streams

Kafka Streams applications read from and write to Kafka only, so they can use Kafka transactions directly. Setting processing.guarantee to exactly_once_v2 (the default is at_least_once) commits the consumed offsets, the output records and the state store changelog in one transaction. It needs brokers at version 2.5 or newer, and it changes the default commit.interval.ms from 30 seconds to 100 ms. The guarantee ends at Kafka's edge: a remote call made from inside the application, such as updating an external store, is not covered.

Idempotence: the tool that always works

Whatever the engine, the last hop to an outside system (a payment API, an email service, a legacy database) usually has no transactions you can join. The fallback is to make that hop idempotent: send a stable ID with every effect and let the receiver ignore an ID it has already applied.

The ID set and the balance must change in one atomic write; if they are stored separately, a crash between the two writes brings back the double count.

What you will be able to do

Explain what exactly-once does and does not promise.
Describe microbatch recovery and checkpoint-barrier recovery, and their latency costs.
Walk through Flink's two-phase commit to Kafka and the settings that keep it safe.
Make a sink idempotent when no transaction is available.

Key concepts

Exactly-once (effect): After failures and replays, the output looks as if each event was processed once. Events may be processed more than once internally.
Microbatching: Running a stream as a sequence of small batches. A failed batch is re-run over the same saved offset range. Spark Structured Streaming does this by default.
Checkpoint: A consistent snapshot of every operator's state plus the source offsets it corresponds to, saved to durable storage. Recovery restarts from the latest completed one.
Checkpoint barrier: A marker injected at the sources that flows in line with records. An operator saves its state when the barrier has arrived on all its inputs.
Unaligned checkpoint: A checkpoint where barriers overtake in-flight records, which are saved with the state. Faster under backpressure, larger checkpoints.
Two-phase commit sink: Output goes into a transaction that is pre-committed at the barrier and committed only when the whole checkpoint completes, or aborted on failure.
Idempotent write: A write that has the same effect however many times it is applied, for example an upsert by key or a write guarded by an event ID.
read_committed: The Kafka consumer isolation level that hides records from open or aborted transactions. Required for exactly-once output to be seen correctly.

When to use it

Choosing delivery guarantees for a pipeline that moves money, counts usage or feeds analytics
Explaining how Flink checkpoints or Spark microbatches recover after a crash
Designing the last hop from a stream processor into a database or external API
Tuning checkpoint intervals and transaction timeouts for a Kafka-to-Kafka job

Interview questions

How does Flink achieve exactly-once end to end with Kafka as both source and sink?
Would you use at-least-once or exactly-once for a pipeline that charges customers for usage?
How do you choose a checkpoint interval?
Your job has exactly-once enabled, yet customers got duplicate emails after a restart. Why?