Video Understanding System

Shots, Transcripts, Chapters and Segment Search

Design the pipeline that turns every uploaded video into structured data (shots, keyframes, a timed transcript, visual labels, chapters) and a search index that can jump a viewer to the right moment.

Overview

To a computer, an uploaded video is a stream of pixels and audio with no structure. A video understanding system adds the structure: where the shots change, what is on screen, what is said and when, and how the video divides into topics. Other products use that output. Search can jump to the moment someone explains indexing, recommendations know what a video is about, captions come from the transcript, and moderation gets flags to review.

The hard part is cost. One minute of 30 frames-per-second video has 1,800 frames, and neighboring frames are nearly identical. Running a large vision model on each frame of every upload is wasteful. The design runs cheap detectors first to find where the content changes, runs expensive models only on chosen keyframes and on the audio, and processes segments of a video in parallel so a long upload is not stuck on one machine.

Questions to ask first

What is the output for? — Search to a moment, chapters, captions, and signals for other teams
How much video? — 1 million uploads a day, 8 minutes on average
How fast? — A 10-minute video fully processed within 5 minutes; transcript first
Which languages? — Many; the speech model identifies the language
Do we recognize people? — No: no face identification; generic labels only
Can results change later? — Yes: new models reprocess old videos in the background

What you will be able to do

Explain why the pipeline runs cheap detectors before expensive models, with numbers.
Detect shot changes from color histograms and avoid false cuts.
Fan a video out into segments and join the results exactly once, safely under retries.
Produce captions and chapters that follow the rules players enforce.
Build search that returns a moment, combining keyword and vector retrieval.

Key concepts

Shot boundary detection
Keyframe sampling
Segment fan-out
Speech recognition with timestamps
Vision embeddings
Chapters
Hybrid search with RRF
Job DAG
Priority queues

Functional requirements

Process each upload into shots with start and end times, and a keyframe per shot
Transcribe speech with word timestamps and the detected language; export captions (WebVTT)
Label keyframes and compute embeddings for visual search
Propose chapters with titles, following the player's chapter rules
Search across videos and return (video, start time, matching text) results
Publish signals to moderation, recommendation and ads teams
Reprocess old videos when a model improves, without blocking new uploads

Non-functional requirements

A 10-minute video fully processed within 5 minutes of upload; transcript first (targets we chose)
Search answers in under 300 ms at p95 (a target we chose)
Every stage retries safely: a segment is never counted twice, a job never finishes twice
Priority: fresh uploads ahead of backfills; backfills use spare and preemptible capacity
Partial results are usable: a transcript is served even if chapters fail

Interview questions

Why not run the vision model on every frame?
How does shot detection work, and what goes wrong?
How do you process a two-hour video in a few minutes?
How do you make sure the job advances exactly once under retries?
How does search find the right 30 seconds?
How do you generate chapters?
How do you handle a model upgrade across millions of old videos?
Where does the GPU budget go, and how do you cut it?