Automated Content Tagging System

Tagging Images for Search

Design a system that tags every uploaded image from a managed list of tens of thousands of tags, sends uncertain tags to people, lets editors add new tags without waiting for a retrained model, and makes the tags searchable.

Overview

A stock-photo library or a marketplace only works if people can find things. Uploaders rarely tag their images well, so the platform tags them itself: 'dog', 'golden retriever', 'beach', 'sunset', 'drone shot'. With millions of uploads a day, a model has to do most of it.

This is hard for three reasons. The tag list is large and changes weekly, as editors add new tags that no model was trained on. A wrong tag is worse than a missing one, because it puts bad results in front of searchers and is hard to notice. And people are a limited resource: a review team can check some tags, never all of them.

Questions to ask first

What content? — Images; text and video are out of scope
How much? — 200 million images today, 2 million uploads a day
How many tags, and who owns them? — About 20,000, in a hierarchy that editors change weekly
How fast must tags appear? — Searchable within a few minutes of upload
Is a wrong tag or a missing tag worse? — A wrong tag: auto-applied tags must be right about 95% of the time (our target)
Is there a human review team? — Yes, with a budget of about 20,000 reviews a day

What you will be able to do

Explain why one image embedding plus cheap per-tag scores beats one giant classifier.
Tag with a brand-new tag the day an editor adds it, and say how it graduates to a trained one.
Choose per-tag cutoffs from a precision target and size them against a review budget.
Model a tag hierarchy and search it, including 'dog' finding golden retrievers.
Re-tag a 200-million-image library when the model changes, without breaking search.

Key concepts

Image embeddings
Zero-shot tagging
Linear probe
Multi-label classification
Calibration
Per-tag thresholds
Human review queue
Tag hierarchy
Backfill
Content-hash dedup

Functional requirements

Accept an image upload and tag it asynchronously
Tag from a hierarchical taxonomy of about 20,000 tags; a child tag implies its parents
Let editors add, rename, move and retire tags; a new tag works the same day without retraining
Return tags with a confidence and whether a model or a person set them
Send uncertain tags to a review queue; reviewers confirm or reject them
Search for images by combinations of tags
Re-tag the library when a model or a tag definition changes

Non-functional requirements

Tags searchable within 5 minutes of upload at p95 (a target we chose)
Auto-applied tags are correct about 95% of the time, measured per tag on labeled samples (our target)
Reviews stay within the budget of about 20,000 a day
A reviewer's decision is never overwritten by a later model run
Uploads keep working if tagging is down; tagging catches up from the queue

Interview questions

How do you tag with 20,000 tags without training a 20,000-class classifier?
An editor adds a tag today. How is it used before anyone labels data?
How do you decide when a tag is applied automatically?
How does searching for dog find a golden retriever?
How do you keep reviewers' decisions from being overwritten?
You have a better model. How do you re-tag 200 million images?
How do you measure tag quality in production?