Language Detection System

Language Identification as a Shared Service

Design a shared service that tells other systems which language a piece of text is in, fast enough to sit on every write path, honest about short and mixed-language text, and safe to update.

Overview

Language detection (also called language identification) answers one question: which language is this text in? Many systems depend on the answer. A support ticket is routed to the right team, a comment goes to the right moderation model, a search query is matched against the right index, a post is offered for translation. So the detector sits on the write path of many services and must be fast, cheap and predictable.

The core technique is well established. Letters appear in characteristic short sequences in each language (English has many 'th' and 'ing', German has 'sch' and 'ei'), so counting character n-grams (runs of n characters) and comparing them with each language's profile works well on a sentence or more. Open tools do this today: langid.py (Lui and Baldwin, 2012) uses naive Bayes over byte n-grams, Google's CLD3 uses a small neural network over n-gram embeddings, and fastText's lid.176 model recognizes 176 languages. The hard parts are short text, mixed text and change management.

Questions to ask first

Who calls it? — Internal services: routing, moderation, search, translation
How much traffic? — About 10 billion texts a day, mostly single texts, some batches
How fast? — Under 10 ms at p99 inside the data center (a target we chose)
How long are texts? — From one word to many pages; many are under a sentence
What should it say when unsure? — und (undetermined), with the probabilities, never a confident guess
Can callers restrict the answer? — Yes: an allowlist of languages and an optional hint (user locale)

What you will be able to do

Explain how n-gram detectors work and run a tiny one.
Use the Unicode script of the text to cut the candidate languages before any model runs.
Handle short and mixed-language text with reliability thresholds, allowlists, priors and spans.
Size the service, and roll out a new model without surprising every caller.

Key concepts

Character n-grams
Naive Bayes
Unicode script
BCP 47 tags
und (undetermined)
Reliability threshold
Allowlists and priors
Mixed-language spans
Shadow rollout

Functional requirements

Return the most likely language as a BCP 47 tag, with a probability and the top 3 candidates
Return und (undetermined) with is_reliable false when the text carries too little signal
Accept an allowlist of languages and an optional hint such as the user's locale
Report spans with their languages for text that mixes languages
Detect batches of up to 1,000 texts in one call
Record accuracy per language and per length bucket from labeled samples

Non-functional requirements

p99 under 10 ms for a single text inside the data center (a target we chose)
Stateless: any pod can serve any request; capacity grows by adding pods
No raw text stored by default; only 1% sampled records for quality checks, with a retention limit
A model change never reaches all traffic before it passes shadow comparison
If the service is down, callers fall back to their own default (for example the user's locale) rather than failing

Interview questions

How does a language detector work?
Why is short text hard, and what do you do about it?
How do you handle text in two languages?
How do you scale it to hundreds of thousands of texts a second?
How do you ship a new model safely?
What language codes should the API return?