Lambda and Kappa Architectures

Lambda runs a batch pipeline and a streaming pipeline side by side and merges their answers; Kappa keeps one streaming pipeline and reprocesses by replaying the log.

Overview

The problem: fresh or correct

Analytics systems want two things at once. They want results that are fresh, covering events from the last few seconds. They also want results that are correct and complete, and that can be recomputed when the code had a bug. Early stream processors were fast but hard to trust; batch jobs were trustworthy but hours behind.

There are two classic designs for this. Lambda runs both kinds of pipeline and combines them. Kappa runs only a streaming pipeline and treats reprocessing as replaying its input.

Pipelines — Batch and streaming, side by side — Streaming only
Code bases — Two, which must agree — One
Reprocessing — Rerun the batch job over all raw data — Replay the retained log through a new job
Origin — Described by Nathan Marz (2011 post; the name came later) — Named by Jay Kreps (2014 article)

Lambda: three layers

Nathan Marz described the idea in 2011: keep an immutable master copy of all raw data, and think of every query as a function of all data. Three layers make that practical.

The batch layer recomputes views from all raw data on a schedule. It is slow, often hours behind, but simple and correct.
The serving layer stores those batch views for fast reads.
The speed layer (Marz called it the realtime layer) processes only recent events, filling the gap since the last batch run. When the next batch run covers those events, the speed layer's results for them are thrown away.

The appeal is human fault tolerance: raw data is never overwritten, so a bad deploy can be fixed by correcting the code and recomputing. Any error in the speed layer lasts only until the next batch run.

What went wrong with Lambda

The cost is that every piece of logic is written twice, for two different systems, and the two versions must produce the same answer. Jay Kreps, then at LinkedIn, argued in 2014 that keeping them in agreement is as painful as it sounds. Google's Dataflow paper (2015) made the same point: Lambda systems meet many requirements but fail on simplicity, because you build and run two systems. It also describes a team whose users stopped trusting the fast, weakly consistent results, which led it to rebuild the system around strong consistency.

Logic drift — A fix lands in the batch job but not the stream job, or the other way round
Double operations — Two clusters to deploy, monitor and debug
Merge logic — The serving query must know where the batch view ends and the realtime view begins
Confusing numbers — A dashboard changes when the batch run replaces the realtime estimate

Kappa: one pipeline, replay to reprocess

Kreps proposed dropping the batch layer. Keep the raw events in a log (Kafka) for as long as you may need to reprocess; he gave the example of 30 days of retention to reprocess 30 days. To change the logic, run a second copy of the job from the start of the log.

The cost moves elsewhere. The log must keep enough history, and the stream processor must replay that history fast enough to be useful.

What made Kappa practical

Kappa depends on engines that treat a bounded dataset (a batch) and an unbounded one (a stream) with the same program.

Unified engines — Apache Flink runs the same DataStream program in BATCH mode on bounded input or STREAMING mode on either; Apache Beam offers one model for bounded and unbounded collections, run on Flink, Spark or Google Cloud Dataflow
Event-time processing — Results grouped by when events happened, so a replay gives the same answer as the live run
Long log retention — Kafka tiered storage (KIP-405) moves completed log segments to remote storage such as S3 or HDFS; early access from Kafka 3.6, production-ready since Kafka 3.9
Exactly-once inside Kafka — Transactions (since Kafka 0.11) commit output and consumer offsets together for Kafka-to-Kafka jobs

Event time, not arrival time

Replay only matches the live run if results depend on when each event happened (event time), not when the processor saw it (processing time). During a replay, every event is seen at nearly the same moment, so processing-time windows would put a month of events into a few minutes. Late events break processing-time windows even in live runs, as this small example shows.

Event time — 4 — 2
Processing time — 3 — 3

The event at second 59 arrived 16 seconds late. Event time still counts it in minute 0; processing time moves it to minute 1.

When batch still earns its place

Kappa is the default choice for new designs, but it is not free. If replaying months of history through the stream job would take longer than you can wait, or the raw data lives in files that were never in a log, a batch job over files is still the practical tool. Many teams run one code base (Flink, Beam or Spark) in both batch and streaming modes, which keeps Lambda's reprocessing without Lambda's second code base.

What you will be able to do

Draw Lambda's batch, speed and serving layers and say what each one does.
Explain why two code bases for one result is Lambda's main cost.
Walk through Kappa reprocessing: replay into a new table, catch up, switch, delete the old one.
Explain why event time is needed for a replay to match the live result.
Say when a batch job is still the right tool.

Key concepts

Lambda architecture: A batch layer over all raw data plus a speed layer over recent data, with queries merging the two views.
Batch layer: Recomputes views from the full, immutable raw dataset on a schedule. Slow but simple and correct.
Speed layer: Processes only events not yet covered by the last batch run. Its results are replaced when the batch catches up.
Serving layer: Stores the batch views and answers reads from them quickly.
Kappa architecture: One streaming pipeline. Reprocessing means replaying the retained log through a new version of the job.
Event time: When an event actually happened, as recorded in the event. Grouping by it makes replays give the same answer as live runs.
Processing time: When the processor handled the event. Late events and replays move events to the wrong window.
Bounded and unbounded input: A dataset with an end (a batch) versus one that keeps growing (a stream). Unified engines run the same code on both.

When to use it

Designing a system that needs both real-time and historical analytics.
Explaining how to recompute results after a bug in a streaming job.
Discussing why teams moved from Lambda to a single streaming code base.
Deciding whether a backfill should run as a replay or as a batch job.

Interview questions

Why did many teams move from Lambda to Kappa?
In Kappa, how do you reprocess after fixing a bug?
What does event time versus processing time mean, and why does it matter here?