Apache Flink Deep Dive
Apache Flink, the stateful stream processor: the runtime, keyed state and state backends, event time and watermarks, windows and late data, checkpoints with barriers, end-to-end exactly-once with Kafka, and savepoints.
Overview
Apache Flink is a framework for running computations over streams of events, keeping state between events, and producing correct results even when events arrive late or machines fail. A Flink job reads from sources such as Kafka, transforms and aggregates events one by one, and writes to sinks such as Kafka, databases or files.
Flink stands apart from simpler stream tools in three ways. It treats state as part of the system: counts, windows and joins keep their data inside Flink, partitioned by key and saved in consistent snapshots. It measures time by event time, the moment each event happened, not when it arrived. And it recovers from failure by restoring the last snapshot and replaying the input from there, which gives exactly-once state.
A bounded dataset, such as yesterday's files, is just a stream that ends, so the same engine runs batch jobs too. Interviewers use Flink to test whether you understand streaming correctness: late data, out-of-order events, recovery, and duplicate output.
What you will be able to do
- Describe how a Flink job runs: JobManager, TaskManagers, slots, parallel tasks and keyed state.
- Explain event time, watermarks and late data, and pick a window type for a problem.
- Walk through a checkpoint with barriers and say what recovery replays.
- Explain what exactly-once means in Flink and what a sink must do to extend it to output.
- Choose a state backend, and say when to use a checkpoint and when a savepoint.
Key concepts
- JobManager and TaskManagers
- Keyed state
- State backends
- Event time and processing time
- Watermarks
- Windows and late data
- Checkpoint barriers
- Exactly-once sinks
- Savepoints
- Backpressure
Interview questions
- What is a watermark, and how do you handle late data?
- How do Flink checkpoints give exactly-once state?
- How do you get end-to-end exactly-once from Kafka to Kafka?
- Which window would you use to count active users per 5 minutes, and to measure user sessions?
- Heap or RocksDB state backend?
- What is the difference between a checkpoint and a savepoint?
- A Flink job's consumer lag keeps growing. How do you debug it?
- When would you choose Flink over Spark Structured Streaming or Kafka Streams?
Sign up free to read the full topic
More System Design topics
Start free on Cariara · Coding problems · System design studio