News Aggregator

Stories from Many Publishers

Design a service that collects articles from tens of thousands of publisher feeds, groups the ones about the same event into stories, and shows each reader a fresh, ranked front page within minutes of publication.

Overview

A news aggregator does not write news. It reads what thousands of publishers post, groups articles that describe the same event into a story, and shows readers a front page of stories with a headline, a snippet and links out to each publisher.

Three problems make it interesting.

Fetching. Publishers expose new articles in feeds (RSS 2.0 or Atom). Checking 50,000 feeds often enough to be fresh, without hammering small sites, is a scheduling problem.
Grouping. A wire story is republished almost word for word by hundreds of outlets, while others write their own versions. Exact copies, near-copies and different articles about the same event all need to land in one story.
Ranking while things change. A story that is ten minutes old and growing fast matters more than a bigger one from yesterday. The front page must reflect that within minutes.

Questions to ask first

Where do articles come from? — 50,000 publisher feeds (RSS 2.0 and Atom); a few push with WebSub
How many? — About 2 million new articles a day
What do we store? — Headline, snippet, link, time, source and signals; not the full article
How many readers? — 50 million a day, about 5 front-page loads each
How fresh? — A new article in its story within about 5 minutes; a breaking story on the front page soon after
Personalized? — A shared top-stories list per region, plus sections ranked by each reader's interests

What you will be able to do

Schedule polling of tens of thousands of feeds with conditional GET, adaptive intervals and push where offered.
Turn RSS and Atom items into one article record with a stable identity.
Detect exact and near-duplicate articles with canonical URLs and SimHash.
Cluster articles into stories online, and rank stories by freshness, coverage and source quality.
Spot a breaking story from how fast articles join it.

Key concepts

RSS 2.0 and Atom
Conditional GET
Adaptive polling
WebSub push
Canonical URLs
SimHash
Shingles and MinHash
Online clustering
Freshness decay
Velocity detection

Functional requirements

Fetch new items from 50,000 publisher feeds and from WebSub pushes
Store each article once, even when it appears in several feeds or is updated
Group articles about the same event into a story, with a representative headline
Top stories per region and language, and per section (world, business, sports, technology)
A personalized front page from followed topics and reading history
A story page listing every source that covered it

Non-functional requirements

An article appears in its story within about 5 minutes of publication (a target we chose)
Front page p99 under about 200 ms (a target we chose)
Polite fetching: honor robots.txt, conditional requests and publisher rate limits
The read side stays up when the pipeline is behind (serve the last good lists)
No single source can flood the front page

Interview questions

How do you keep 50,000 feeds fresh without hammering publishers?
How do you tell that two articles are the same?
How does story clustering work, and how do you fix its mistakes?
How do you rank the top stories?
How do you personalize without losing the big news?
How do you detect breaking news?
What happens when a publisher offers WebSub?