IoT Data Ingestion Pipeline

Millions of devices, telemetry in, commands out

Design the pipeline that connects millions of sensors and machines over MQTT, takes in their readings without losing or double-counting them, stores them as time series, and sends commands back, even to devices that are asleep.

Overview

An IoT pipeline connects physical devices (thermostats, meters, cold-chain sensors, factory machines) to the cloud. Devices send small readings every few seconds; the platform stores them, raises alerts, and sends commands back.

The devices make it hard. Many are small, run on batteries, and sit on flaky cellular or Wi-Fi links. They lose connection, buffer readings, reconnect and resend. Their clocks drift. Some sleep for hours and cannot be reached. And there are millions of them, each holding a connection open.

MQTT is the usual protocol for this. It is an OASIS standard (version 5.0 is current): devices keep one TCP or TLS connection to a broker, publish messages to topics such as devices/42/telemetry, and subscribe to topics for commands. The broker routes messages between publishers and subscribers; neither side needs to know the other.

Questions to ask first

How many devices, how often? — 10 million connected, one message every 10 seconds each
How big is a message? — About 200 bytes: 5 readings with a timestamp
Can a reading be lost? — Routine readings: occasionally. Alarms and commands: never
How long is data kept? — Raw for 30 days, one-minute rollups for 2 years
Do we command devices? — Yes, including ones that are offline or asleep
How are devices identified? — A certificate per device, installed at manufacture

What you will be able to do

Explain MQTT QoS 0, 1 and 2, and pick one per message type.
Count every reading once even though devices resend, and keep each device's readings in time order.
Apply backpressure from the database all the way to the device, with a clear rule for what gets dropped.
Store billions of readings a day as time series with rollups and retention.
Deliver a command to a device that is asleep, using desired and reported state.

Key concepts

MQTT 5.0
QoS 0, 1, 2
Persistent sessions
Device certificates
Topic ACLs
Receive Maximum
Deduplication by message ID
Event time vs arrival time
Time-series rollups
Desired and reported state

Functional requirements

Register devices with a unique identity and let them connect securely
Accept telemetry over MQTT and route it to storage and rules
Store readings as time series and answer range queries per device and metric
Evaluate alert rules on incoming readings within seconds
Set a desired state for a device and deliver it when the device next connects
Show each device's connection status and last-seen time

Non-functional requirements

Alarms and commands are never lost once acknowledged; routine readings may be dropped only under overload
Each reading is stored once, even when the device resends it
Median time from publish to stored, under 5 seconds (a target we chose)
Alerts fire within 10 seconds of the reading that triggers them
A device can only publish and subscribe to its own topics
A slow database never causes devices to be disconnected

Interview questions

Why MQTT rather than HTTP for devices?
Which QoS would you use, and why?
How do you make sure each reading is counted once?
The time-series database is down for an hour. What happens?
How does backpressure reach the device?
How do you send a command to a device that is asleep?
How do you know a device is offline?
How is the broker cluster scaled?