Price Comparison Engine

Collect, Match and Track Retail Prices

Design a service that collects prices from many retailers, recognizes when two listings are the same product, keeps each product's price history, and alerts users when a price falls to their target.

Overview

A price comparison engine shows, for one product, what each retailer charges, and how that price has moved. Users search for a product, compare offers, set an alert for a target price, and click through to buy on the retailer's site.

The design has three problems to solve. Getting prices: the best source is a feed the retailer sends; the fallback is fetching product pages, politely, within a limited budget. Knowing it is the same product: 'Acme AB-100 wireless headphones, black' on one site and 'ACME AB100X' on another must land on one product page, and two different items must never be merged. Remembering prices: history is what tells a user whether today's sale is real.

Questions to ask first

How do we get prices? — Retailer feeds where offered; otherwise crawl public product pages that robots.txt allows
How many? — 2,000 retailers, 10M products, 100M retailer listings
How fresh? — Popular listings hourly, the rest daily or weekly
Which price do we compare? — Landed price: item price plus shipping, in one currency
How long is history kept? — At least a year, at change granularity
Do users buy on our site? — No; they click through to the retailer

What you will be able to do

Choose between feeds and crawling, and crawl within robots.txt and a per-site rate.
Split a limited fetch budget across listings by popularity and volatility.
Normalize GTINs and match listings to products without false merges.
Store price history as change rows and answer range questions quickly.
Fire each price alert exactly once when a price change satisfies it.

Key concepts

Merchant feeds vs crawling
robots.txt (RFC 9309)
Crawl budget and tiers
Structured data extraction
GTIN normalization
Product matching
Landed price
Change-only price history
Event-driven alerts

Functional requirements

Collect price, shipping, currency and stock for each retailer listing, from feeds or crawled pages
Match listings to a single product across retailers
Show a product page with every retailer's landed price and when it was last checked
Keep price history per listing and show the low, high and usual price over a period
Let users set a target-price alert and notify them once when it is met
Search products by keyword, brand, category and price

Non-functional requirements

Respect robots.txt and a per-site request rate; never evade blocks (a policy we chose)
Popular listings are refreshed within about an hour; each price shows its last-checked time
No false merges of different products: when unsure, keep listings separate
Alerts are sent within minutes of the price change being seen, and never twice for one alert
Search answers in under 300 ms at p99 (a target we chose)

Interview questions

How do you get prices from thousands of retailers?
How do you crawl without getting blocked?
How do you know two listings are the same product?
How do you store price history efficiently?
How do you fire price alerts at scale, exactly once?
How do you decide what to crawl next?
How do you tell a real discount from a fake one?