Document AI Processing System

OCR, layout and field extraction with human review

Design a service that takes scanned and digital documents (invoices, receipts, forms), reads the text, finds the fields and tables that matter, checks them, and sends the uncertain ones to people, at millions of pages a day.

Overview

Companies receive documents they need as data: invoices to pay, receipts to reimburse, forms to file. Many arrive as scans or phone photos, so there is no text to read, only pixels. A document AI system turns each one into structured fields (vendor, date, total, line items) that software can use, with a confidence for each field and a record of who changed what.

The pipeline has four jobs. Read the text: from the PDF's text layer when it has one, otherwise with OCR (optical character recognition). Understand the layout: which words form a table, which label belongs to which value. Extract and check the fields the document type needs. Route: accept what is certain, send the rest to people, and learn from their fixes.

Interviewers care less about the OCR model than about the system around it: how work flows asynchronously through many stages, how confidence and business rules decide what people see, and how the design stays correct when jobs are retried.

Questions to ask first

Which documents? — Invoices, receipts and forms; mostly English; a mix of digital PDFs and scans
How much? — 10 million pages a day, about 4 pages a document, busiest in business hours
How fast? — Results within minutes; a one-page upload can be answered in seconds
How accurate? — Money fields must be right; a person checks anything the system is not sure of
Who reviews? — The customer's own staff, in our review tool
Keep documents? — Yes, originals and every correction, for audit

What you will be able to do

Lay out an asynchronous, page-level pipeline and size it from pages a day.
Explain OCR, layout analysis and table reconstruction, and when each is needed.
Route fields with per-field thresholds and validation rules, and size the review team.
Build a review queue many people can work at once, with a full audit trail.
Keep results reproducible across model upgrades and retries.

Key concepts

OCR
Text layer vs scan
Layout analysis
Table reconstruction
Key-value extraction
Document classification
Confidence thresholds per field
Validation rules
Human-in-the-loop review
Model versioning

Functional requirements

Accept PDF, TIFF, PNG and JPEG by upload, batch URL list or email
Read text from the PDF text layer when present, otherwise by OCR, including handwriting where supported
Classify each document (invoice, receipt, form) and extract that type's fields, tables and checkboxes
Return each field with its value, confidence, page and bounding box
Check fields with rules (totals add up, dates are valid) and send uncertain fields to human review
Let reviewers correct fields; keep the original value, the new value, who and when
Notify the customer by webhook when a document is done

Non-functional requirements

Accepted documents are never lost: stored before the upload call returns
A 10-page document is done within 2 minutes at p95, excluding human review (a target we chose)
Money fields accepted without review are wrong less than 1 time in 1,000 (a target we chose, measured by sampled audits)
Retries never create duplicate documents, fields or webhooks
Every result records the model version that produced it

Interview questions

Walk me through what happens to one uploaded invoice.
How do you decide which fields a person must check?
How do you extract tables?
How big is the review team, and how do you keep it from growing with volume?
How do many reviewers work the same queue without conflicts?
What makes this pipeline safe to retry?
How do you ship a better extraction model without breaking audits?
Would you use a hosted service like Textract or your own models?