Real-Time Event Stream Processing

A stateful pipeline from ingest to sink

Design a pipeline that counts a high-rate stream of card transactions over time windows and raises alerts within seconds, with correct results despite late events, failures and replays.

Overview

A card network sends every transaction as an event. The business wants two things from that stream: an alert when one card is used too often in a short time (for example more than 5 times in 10 minutes), and a per-merchant count and total every minute for a live dashboard. Both are questions over time, asked continuously, over data that never ends.

This problem is the interview walk-through for any stateful streaming pipeline. You go stage by stage: how events get in, how they are split across machines, where the running counts live, how windows decide when a minute is over, what happens to an event that shows up late, and how the output stays correct after a crash or a replay. The concepts have their own topics (Batch vs Stream Processing for time and windows, the Apache Flink Deep Dive for one engine's internals, Delivery Semantics for exactly-once, and the Ad Click Event Aggregation problem for a counting pipeline end to end); here the goal is to put them together in the order an interviewer expects.

Questions to ask first

How many events? — 200,000 a second at peak, 100,000 on average, about 1 KB each
How fresh must results be? — Alerts within 5 seconds; dashboard within a minute
Does order matter? — Per card, yes; across cards, no
How late can events be? — Most within seconds; some (offline terminals) minutes or hours
What if a count is wrong? — Alerts may be corrected later; the daily total must be exact
Can we replay? — Yes: keep the raw events for 7 days

What you will be able to do

Walk a streaming design in the order interviewers expect: ingest, partition, state, windows, late data, sinks, operations.
Choose partition keys and size parallelism from the event rate.
Explain watermarks, allowed lateness and side outputs with a worked example.
Make results exactly-once with checkpoints plus a transactional or idempotent sink.
Handle hot keys, lag and replays.

Key concepts

Partitioned log
Partition key
Keyed state
Tumbling and sliding windows
Watermark
Allowed lateness
Side output
Checkpoint
Transactional or idempotent sink
Consumer lag

Functional requirements

Ingest transaction events from many producers without losing any
Count transactions per card over a 10-minute window, sliding every minute, and alert above a threshold
Compute per-merchant count and total per one-minute window for the dashboard
Use event time; accept late events for a set time and correct results
Keep events that are too late for review instead of dropping them silently
Replay a time range after a bug fix and overwrite the affected results

Non-functional requirements

Alerts within 5 seconds of the event at p99 (a target we chose)
No event lost; no event counted twice in any sink, across failures and replays
Recover from a lost worker in under a minute, from the last checkpoint
Scale out by adding partitions and workers, without changing results
Lag, checkpoint time and late-event counts are visible and alerted on

Interview questions

Walk me through the design from event to result.
How do you choose the number of partitions?
What is a watermark, and what happens to events that arrive after it?
How do you get exactly-once results?
One merchant produces a third of all events. What breaks, and how do you fix it?
How do you deploy a fix and correct two hours of bad results?
How is this different from the Ad Click Event Aggregation problem?