Voice Assistant Backend

Speech to intent to action

Design the cloud side of a voice assistant: stream audio after the wake word, turn speech into an intent, keep multi-turn dialog state, call a skill and speak the answer within about 1.5 seconds.

Overview

A voice assistant turns a spoken request ('set a timer for ten minutes') into an action and a spoken reply. The device listens locally for a wake word; only after it hears one does it open a stream to the cloud. The backend then runs a pipeline: speech recognition (ASR, audio to text), language understanding (NLU, text to an intent such as set_timer with slots such as duration = 10 min), dialog management (what the conversation already knows), a skill that does the work, and speech synthesis (TTS, text back to audio).

The user is waiting in silence, so latency is the product. The clock that matters starts when the user stops talking and ends when the reply starts playing. Two parts of that time are easy to miss: the system must first *decide* that the user has stopped, which means waiting for some silence, and the skill may belong to a third party whose speed you do not control.

Voice is also more ambiguous than typing. People say 'make it large' or 'cancel that', leave out details ('set a timer'), and pause in the middle of a sentence. And the audio is private: it comes from people's homes.

Questions to ask first

How many devices, how many requests? — 100 million active devices, about 5 requests each a day, peaks at 3 times the average
What latency? — 1.5 s at p95 from end of speech to first reply audio, a target we chose
Where does the wake word run? — On the device; audio goes to the cloud only after it fires
Multi-turn conversations? — Yes: follow-ups, clarifying questions and references like 'it'
Third-party skills? — Yes, hosted by their developers and called over HTTPS
What do users control? — They can list, replay and delete their voice history, and set a retention period

What you will be able to do

Split a 1.5 second budget across endpointing, recognition, understanding, the skill and speech synthesis.
Explain why the wake word runs on the device, and how a second check in the cloud cancels false wakes.
Design endpointing and explain what the silence timeout costs.
Keep multi-turn dialog state so 'make it large' and 'cancel that' work.
Call third-party skills with timeouts, and store voice history so a deletion really deletes.

Key concepts

On-device wake word
Streaming ASR
Endpointing
Intents and slots
Dialog state
Reference resolution
Skill routing with timeouts
Progressive responses
Streaming TTS
Voice history deletion

Functional requirements

Accept a streaming audio session from a device after its on-device wake word fires
Return partial transcripts while the user speaks and a final transcript when they stop
Extract an intent and its slots, ask for missing slots, and resolve references to earlier turns
Route the intent to a built-in or third-party skill and speak its answer
Cancel the session if the cloud decides the wake word was a false alarm
Let users list, replay and delete their voice history, and set a retention period

Non-functional requirements

End of speech to first reply audio in 1.5 s at p95 (a target we chose)
Audio travels encrypted and compressed; raw audio is never streamed before the wake word
A slow or failing third-party skill never blocks the session past its timeout
Dialog state survives a gateway restart within a conversation (kept outside the gateway)
A deletion request removes the transcript, the audio and derived copies within a stated time

Interview questions

Walk me through the latency budget.
Why not detect the wake word in the cloud? It would be more accurate.
How do you know the user has finished speaking?
How does 'make it large' work?
A third-party skill is slow. What happens?
How do you make deletion real?
How would you scale this to many regions?