Manufacturing Platform: Five Nines Across Plants and Clouds

MES, vision inspection and telemetry that survive a lost WAN or cloud

Design a multi-plant manufacturing platform (MES, AI vision, telemetry, predictive maintenance, ERP) on AWS, Azure and Google Cloud that keeps a 99.999% production SLA when a plant loses its WAN or a cloud fails.

Overview

A manufacturer runs dozens of plants. Each plant has production lines full of machines controlled by PLCs (programmable logic controllers). On top of them run the services this design is about: the MES (manufacturing execution system) that dispatches production orders to lines and records what was built from what (genealogy); quality inspection, where cameras and AI models accept or reject parts; line telemetry from thousands of sensor tags; predictive maintenance that scores machine health; and ERP integration that brings orders in and posts what was produced back to the business systems.

The business asks for a 99.999% SLA on production: about 5 minutes a year in which the platform stops a line. That number decides the whole design. A WAN link, a cloud region or a cloud failover can each lose more than 5 minutes in a single bad year, so production cannot depend on any of them. The plant must keep producing with the WAN cut and every cloud down, and the cloud tier must accept that it is hours behind sometimes and catch up without losing or doubling data.

This topic is about what is specific to manufacturing: the plant tier, the IT/OT boundary, store-and-forward, edge AI and per-tier recovery. The generic multi-cloud architecture is in the Microservices at 99.999% on AWS, Azure and Google Cloud topic and the DR program in the Business Continuity and Disaster Recovery Across Clouds topic; both are referred to, not repeated.

Scope it first

What exactly is the 99.999%? — Minutes a line is stopped because of the platform, per plant per year; machine faults are not counted
How many plants and lines? — 40 plants, 6 lines each, 2,000 sensor tags a line sampled once a second
How long can a plant lose its WAN? — Design for 72 hours without loss; longer outages degrade analytics, never production
Which clouds, and why several? — AWS, Azure and Google Cloud, each plant homed to one; acquisitions and regional rules brought them in
Is any cloud in a control or safety loop? — No. Safety stays in PLCs and safety systems; the cloud only advises
Vision inspection? — 4 cameras a line, a part a second, a verdict needed within the line's cycle
ERP? — One ERP of record reached through the cloud; the plant holds released orders ahead

The interviewer listens for four things. Do you derive that five nines rules out any cloud dependency for production, with numbers? Do you draw the plant as its own system with a clear IT/OT boundary? Do you size the buffer for a WAN outage and explain how replay stays exactly-once? And do you keep safety and control loops entirely out of the cloud?

Key concepts

ISA-95 levels
IT/OT boundary and DMZ
Zones and conduits
Plant tier 0
Store-and-forward
Idempotent replay
OPC UA and MQTT
Edge AI rollout
Composite availability
Home cloud per plant

Functional requirements

Dispatch released production orders to lines and record genealogy (which serial was built from which lots) at every station
Inspect every part with AI vision at the line and hold or reject failures within the line cycle
Collect line telemetry from OPC UA servers into a plant historian and forward it to the cloud
Score machine health for predictive maintenance in the cloud and push scores and models back to plants
Receive orders from the ERP and post production confirmations and material consumption back to it
Keep all plant functions running with the WAN down and replay what was produced once it returns
Roll out new software and AI models to plants in stages, with automatic rollback

Non-functional requirements

99.999% availability of the plant production path (our target): at most about 5.3 minutes a year of platform-caused line stops
No telemetry, inspection or genealogy record lost through a WAN outage of up to 72 hours (our choice)
Replay is idempotent: a record delivered twice is stored once, and per-stream order is kept
Vision verdict within 200 ms of the image (our target, inside a 1 s line cycle)
Cloud tier: RTO 1 hour and RPO 15 minutes for tier 1 services, 24 hours for tier 2 (our choices)
No inbound connection from IT or the cloud into the control network; all plant traffic to the cloud is started from inside
No cloud service in any safety function or closed control loop

Interview questions

How can a platform that spans three clouds promise 99.999% for production?
Walk me through what happens when a plant loses its WAN for six hours.
Where is the IT/OT boundary, and what may cross it?
Which edge runtime would you use, and what would you change from its defaults?
How do you make replay exactly-once when MQTT QoS 1 is at-least-once?
How do you roll out a new vision model to 240 lines safely?
What does the second cloud actually do for a manufacturer?
Why must no cloud be in a safety or control loop?