Real-Time AI Translation System

Live Speech Translation

Design a system that listens to live speech and shows (or speaks) a translation within a second or two, keeping each conversation's context and terminology, for hundreds of thousands of conversations at once.

Overview

A live translation system listens to someone speak and shows the words in another language while they are still talking, like subtitles for a conversation. It is three models in a row (speech recognition, machine translation and, optionally, speech synthesis) wrapped in a system that must answer in about a second.

The hard part is timing. Speech recognition keeps changing its mind about the last few words as more audio arrives ('hyper tension' becomes 'hypertension'). Translation wants whole sentences, and some languages put the verb at the end, so translating too early produces the wrong sentence. Waiting for the end of every sentence is accurate but feels slow. The system therefore decides, many times a second, which words are stable enough to translate and which to show as tentative.

Questions to ask first

Captions, voice, or both? — Captions first; spoken output is optional
One speaker or a conversation? — Two-way conversations and meetings; each speaker is one audio stream
How many at once? — 200,000 concurrent audio streams at peak
How fast? — Committed caption within 1.5 s of the word being spoken, p95 (a target we chose); tentative words sooner
Which languages? — Dozens of pairs; the source language may be detected
Specialist words? — Customers upload glossaries (medical, legal, product names)
Keep transcripts? — Only if the customer opts in

What you will be able to do

Lay out the pipeline and give every stage a latency budget.
Decide when a word is stable enough to translate, and code the rule.
Compare wait-k, re-translation and waiting for pauses, and say what each costs.
Keep a session's state on one host, and survive reconnects and host failures without duplicate or missing lines.
Enforce customer terminology with glossaries and size the GPU fleet.

Key concepts

Streaming speech recognition
Voice activity detection
Stable prefix (LocalAgreement)
Wait-k policy
Re-translation and flicker
Session affinity
Glossaries and constrained decoding
Latency budget
Stream rotation

Functional requirements

Stream microphone audio and receive translated captions as the person speaks
Show tentative words at once and replace them with committed words when they are stable
Optionally speak the translation in the target language
Detect the source language or accept it from the user
Apply a customer glossary so chosen terms are always translated the same way
Resume after a dropped connection without losing or repeating captions
Save the bilingual transcript when the customer opts in

Non-functional requirements

Committed caption within 1.5 s of the spoken word at p95; tentative words within about 0.5 s (targets we chose)
Captions never go backward once committed
Audio stays in the region where the session started
A session host failure costs a pause of a few seconds, not the session
Graceful degradation: show source-language captions if translation is down

Interview questions

Walk me through the latency budget.
How do you decide when a word is final?
Why not translate every partial result as it arrives?
How do you handle languages with different word order?
Where does session state live, and what happens when a host fails?
How do you make sure a drug name is always translated the same way?
How many GPUs, and how do you keep them efficient?
How do you degrade under load or partial failure?