AI Content Moderation at Scale

Classify, Review, Appeal

Design a system that checks every post, image and video against content policy, removes known bad content at once, sends uncertain cases to people in order of likely harm, and handles appeals.

Overview

A content moderation system checks what users post against the platform's rules (its content policy) and takes an action: leave it up, remove it, limit who sees it, or ask a person to decide. It runs on every upload, on reports from users, and again when a policy changes.

The shape of the problem drives the design. Almost everything posted is fine, so checking it must be cheap. A small share is harmful, and its harm grows with each view, so the dangerous cases must be caught early and reviewed first. Some content has been seen before (the same illegal image uploaded again), and can be caught by matching a fingerprint instead of running a model. Mistakes go both ways: missing harmful content hurts people, while removing legitimate speech hurts users and, under laws such as the EU Digital Services Act, has to be explained and open to appeal.

Questions to ask first

What content? — Text posts and comments, images, and videos up to a few minutes
How much? — 1 billion new items a day: 700M text, 250M images, 50M videos
Block the upload or check after? — Text and images are checked before they are shown; video is published after a quick check and fully scanned within minutes
What actions? — Allow, remove, limit reach, send to review; always with a reason code
Where do people come in? — Uncertain cases, appeals, and new kinds of abuse
Which laws? — EU DSA duties for statements of reasons and complaints; US duty to report child sexual abuse material to NCMEC

What you will be able to do

Lay out a tiered pipeline and estimate how much reaches each stage.
Match re-uploads of known bad images with perceptual hashes, and search them by Hamming distance.
Keep thresholds as policy per category and market, separate from the models.
Order the review queue by expected harm and hand out work safely to many reviewers.
Handle appeals, explanations and legal reporting, and measure how much harmful content is actually seen.

Key concepts

Perceptual hashing
Hamming distance
Tiered classifiers
Policy thresholds
Review priority by expected harm
Leased work queue
Appeals
Statement of reasons
Prevalence
Evasion

Functional requirements

Check every new text, image and video item, and re-check items when reported or when policy changes
Match uploads against lists of known violating media by perceptual hash
Score items per policy category and apply thresholds that vary by category and market
Queue uncertain items for human review, ordered by expected harm, with specialist queues
Record every decision with a reason code and the policy version, and notify the author
Let authors appeal; appeals are decided by a person

Non-functional requirements

Text and image checks finish in under 300 ms at p99 before the item is shown (a target we chose)
Video fully scanned within 10 minutes of upload (a target we chose)
Severe categories reviewed within 1 hour, others within 24 hours (targets we chose)
No item is reviewed by two people at once by accident, and no claimed item is lost if a reviewer leaves
Every action is auditable: who or what decided, under which policy version

Interview questions

Why a tiered pipeline instead of one strong model?
How do you catch re-uploads of known bad images?
How do you set thresholds, and why not in the model?
How do you order the human review queue?
How do many reviewers take work without collisions?
What do appeals and the law require of the design?
How do you moderate video and live streams?
How do you know moderation is working?