Data Labeling and Annotation System

Human labels at scale, with quality you can measure

Design a labeling platform that gets several independent human labels per item, scores annotators with hidden gold questions, and combines the votes into labels of known quality.

Overview

Supervised models learn from labels: a person says this image shows a cat, this sentence is negative, this box surrounds a pedestrian. A data labeling system turns a pile of unlabeled items into a labeled dataset by giving small tasks to many people, then checking and combining their answers.

It is harder than a web form because people make mistakes, disagree on unclear items, get tired, and sometimes click at random to finish faster. A single label per item gives you no way to tell a careful annotator from a careless one, or a hard item from a bad instruction. The platform's real product is not labels but labels with a known quality.

Three mechanisms do most of the work. Redundancy: several people label each item independently. Gold questions: items with a known answer, mixed in unseen, that score each annotator. Consolidation: combining the labels into one answer, weighted by how good each annotator has proven to be. A model can then take the easy items off human hands (pre-labeling), and choose which unlabeled items are worth a person's time (active learning).

Questions to ask first

What kind of labels? — Image and text classification first; boxes and spans later
Who labels? — A managed pool of about 2,000 annotators online at peak, some with expert skills
How many items? — About 10 million a year
How good must labels be? — Measured per project; 3 labels per item and 5% gold questions by default
Is there a model to help? — Yes, after the first batch is labeled
Can the data leave our systems? — No: annotators see items through signed, short-lived URLs

What you will be able to do

Hand out tasks with leases so no item is lost or labeled twice by the same person.
Choose how many annotators per item, and explain the diminishing returns.
Score annotators with hidden gold questions and weight their votes.
Tell a bad annotator from a bad guideline using agreement metrics.
Use a model to pre-label and to pick the items worth labeling, without training it on its own mistakes.

Key concepts

Task lease
Redundancy
Gold question
Consensus
Weighted vote
Cohen's kappa
Active learning
Pre-labeling
Guideline version
Dataset snapshot

Functional requirements

Create a project with a label schema, instructions (versioned), labels needed per item and gold-question rate
Upload items as references to files in object storage, with optional model pre-labels and priorities
Give each annotator the next task they are qualified for, with a time-limited lease
Accept a label, record the guideline version and time spent, and score it at once if it was a gold question
Combine an item's labels into one answer with a confidence, and route disagreements to an expert
Report per-project and per-annotator quality: gold accuracy, agreement, throughput
Export an immutable, versioned snapshot of consolidated labels

Non-functional requirements

Next-task assignment under 200 ms at p99, so annotators never wait (a target we chose)
Available 99.9% during working hours (a target we chose); a lost lease costs one re-assignment, never a lost label
No annotator labels the same item twice, and gold answers are never sent to the browser
Every label is traceable to its annotator, guideline version and time
An export is reproducible: the same snapshot id always yields the same file

Interview questions

How do you make sure labels are good without reviewing all of them?
How many annotators per item?
How do you hand out tasks to thousands of annotators without conflicts?
How do you tell a bad annotator from a bad guideline?
Where does active learning fit, and what can go wrong?
How do you make a labeled dataset reproducible?
What limits this system at scale?