Speech-to-Text System

Streaming and Batch Transcription

Design a speech-to-text service that turns live audio into text while the person is still talking, and transcribes long recorded files in bulk, with word timings, speaker labels and custom vocabulary.

Overview

A speech-to-text service does two jobs that look alike and are built differently. Streaming: a phone call, a meeting or a voice command sends audio as it happens, and text must come back within a second or so. Batch: a recorded file (a podcast, a two-hour meeting) is uploaded, and the transcript is ready minutes later. Both return words with start and end times, a confidence score and, when asked, which speaker said them.

The model itself is not the design problem in the interview; you can assume one exists. The system problems are the ones around it: splitting endless audio into pieces the model can handle, showing text before a sentence is finished without confusing the reader, keeping hundreds of thousands of live connections each tied to the GPU that holds its state, and cutting long files into pieces that many GPUs can process at once.

Questions to ask first

Live, recorded, or both? — Both: live streams and uploaded files
How many live streams, how much recorded audio? — 200,000 streams at peak; 50,000 hours of files a day
How fast must live text appear? — Partial text while speaking; the final text within about a second of a pause (our target)
Languages? — Several, detected or given by the caller
Speaker labels and custom words? — Yes, both optional per request
Can audio and transcripts be stored? — Only as long as the customer allows; transcripts may be sensitive

What you will be able to do

Explain voice activity detection and endpointing, and pick the silence threshold with its latency cost.
Show partial results that change and final results that do not, and explain stabilization.
Route live streams to stateful GPU workers and survive a worker failure.
Cut long files into chunks, process them in parallel, and assemble the transcript.
Measure accuracy with word error rate and add custom vocabulary without retraining.

Key concepts

Voice activity detection
Endpointing
Partial and final results
Streaming models (CTC, RNN-T)
Sticky sessions
Chunked batch jobs
Diarization
Contextual biasing
Word error rate

Functional requirements

Stream audio over a WebSocket and receive partial and final transcripts
Upload a file (or give a storage URL) for batch transcription and fetch the result when done
Word-level start and end times and a confidence score per word
Optional speaker labels (who spoke when)
Optional custom vocabulary: product names, drug names, jargon
Automatic language detection, or a language given by the caller
Punctuation and casing in final results; optional redaction of numbers such as card numbers

Non-functional requirements

Live: partial text within about 500 ms of speech; final text within about 1 s of the end of a sentence (targets we chose)
Batch: a one-hour file done in minutes by processing chunks in parallel
A worker failure costs a live stream a reconnect, not the transcript so far
Accuracy tracked as word error rate per language and audio type, on a fixed test set
Audio and transcripts encrypted and deleted on the customer's schedule

Interview questions

How do you show text while the person is still talking?
How does the service decide a sentence has ended?
Why not use Whisper for live transcription?
How do you route 200,000 live streams to GPUs?
How do you transcribe a four-hour file quickly?
How would you support custom vocabulary like drug names?
How do you measure accuracy, and what is a good number?