Batch vs Stream Processing
Processing data in scheduled chunks or continuously as it arrives: latency, cost and correctness, event time and watermarks, windows, exactly-once, and the Lambda and Kappa architectures.
Overview
Batch processing runs a job over a fixed set of data, such as yesterday's orders, then stops. Stream processing runs all the time and handles each event soon after it arrives. The difference is not the tools but the data: a batch job sees bounded data with a known end, while a stream job sees unbounded data that never ends, so it must decide when it has seen enough to produce a result.
Batch is the simpler of the two. The input is complete, a failed job can just be run again, and compute is used only while the job runs. Its cost is latency: results are as old as the batch interval. Streaming gives results in seconds, but it must cope with events that arrive late or out of order, keep state for days without losing it, and run and be watched all day.
Most real platforms use both: streams for what must be fresh (fraud checks, alerts, live counters) and batch for what must be complete and cheap (billing, reports, model training), often over the same event log.
What you will be able to do
- Choose batch, stream, or both, from a latency requirement and the cost of being wrong.
- Explain event time, processing time and watermarks, and what happens to late events.
- Pick tumbling, sliding or session windows for a given question.
- Say what exactly-once really guarantees in Flink and Kafka, and what the sink must do.
- Compare the Lambda and Kappa architectures and say how each reprocesses history.
Key concepts
- Bounded and unbounded data
- Event time and processing time
- Watermarks
- Tumbling, sliding and session windows
- Late data
- Exactly-once state
- Backpressure
- Lambda architecture
- Kappa architecture
Interview questions
- When would you choose batch processing, and when stream processing?
- What is a watermark, and what happens to late events?
- Explain tumbling, sliding and session windows, with a use for each.
- How do you get exactly-once results from a streaming pipeline?
- Compare the Lambda and Kappa architectures.
- A consumer cannot keep up with its input. What happens, and what do you do?
- Design a pipeline that shows live ad-click counts and bills advertisers exactly.
Sign up free to read the full topic
More System Design topics
Start free on Cariara · Coding problems · System design studio