Ad Click Event Aggregation

Google Ads

Design a pipeline that counts ad clicks per ad and per minute, fast enough for live advertiser dashboards and exact enough to bill advertisers from.

Overview

When someone clicks an ad, the ad network charges the advertiser (in the cost-per-click model). This system counts those clicks: per ad, per campaign, per minute. Advertisers watch the counts on a live dashboard, and at the end of each day the same counts become an invoice.

That double use is the whole problem. A dashboard can be a minute behind and slightly off; an invoice cannot be off at all. A click must never be lost (lost revenue) and never counted twice (overcharging), yet a distributed pipeline naturally produces both: retries duplicate events, crashes replay them, networks deliver them late. Invalid clicks, from bots or from people inflating an advertiser's costs, must not be billed at all.

The streaming building blocks (windows, watermarks, checkpoints, exactly-once delivery) have their own topics: Batch vs Stream Processing, Apache Flink Deep Dive and Delivery Semantics. This page is how to assemble them for money.

Questions to ask first

How many clicks? — 1 billion a day, peaks of 5× the average
What do we count? — Valid clicks per ad per minute; campaign and advertiser totals roll up from those
How fresh must dashboards be? — About a minute behind
How exact must billing be? — Exact, against the raw click log; corrections are allowed until the invoice closes
How late can a click arrive? — Most within seconds; some (offline phones, retried uploads) hours later
Who decides a click is invalid? — Inline rules now, slower models later; later findings become credits

What you will be able to do

Explain where duplicates and lost clicks come from, and remove each one.
Get exactly-once counts from checkpointed state and an idempotent sink.
Handle late clicks with event time, watermarks and allowed lateness.
Keep a viral ad from overloading one worker.
Reconcile streaming counts against a batch recount and bill from the reconciled numbers.

Key concepts

Click log
Event ID dedup
Exactly-once state
Idempotent sink
Event time
Watermarks
Allowed lateness
Hot keys
Invalid clicks
Reconciliation

Functional requirements

Record every click durably before redirecting the user to the advertiser's page
Count valid clicks per ad per minute; roll up to hours, days, campaigns and advertisers
Show advertisers near-live counts and spend
Remove duplicate events and invalid clicks; credit invalid clicks found after billing
Produce exact daily billing totals that can be traced back to individual clicks

Non-functional requirements

No click lost once the click server has acknowledged it
Billing totals exact against the deduplicated raw log
Dashboards about 1 minute behind real time (a target we chose)
Absorb peaks of 5× the average click rate and single ads that go viral
Clicks keep being recorded even when the aggregation pipeline is down

Interview questions

How do you make sure each click is counted exactly once?
Why not keep a counter per ad and increment it on each click?
How do you handle clicks that arrive late?
How do you handle a viral ad?
Why keep a batch job if the stream job is exactly-once?
How do you deal with click fraud?
Could you use the metrics monitoring system to count clicks?
Where do dashboard queries go, and how fast are they?