CI for Large Models on Custom AI Accelerators

Qualify every layer of the stack on scarce silicon, and know which change broke what

Design the CI/CD system that qualifies large models on a company's own AI accelerator while every layer of the stack changes daily, from correctness and performance checks to GitOps releases on a small accelerator pool.

Overview

A company builds its own AI accelerator and runs large language, vision, speech and recommendation models on it. The software stack between a model and the silicon has many layers, and every one of them changes every day: model code, the framework integration, the graph compiler that lowers a model to the chip, hand-written kernels, the runtime and collective communication library, the kernel driver, the firmware on the chip, and, every year or two, the hardware revision itself. A change in any layer can make a model slower, or quietly make its answers wrong.

You are asked to design the CI/CD system that qualifies models on this silicon. It must catch correctness regressions and performance regressions, decide when a combination of layers is fit to release, promote that combination from development to a staging cluster to the production fleet, and roll it back when something escapes. The hard constraint is capacity: the accelerators that run tests are the same scarce chips the company sells or trains on, so CI gets a small pool and must spend it wisely.

The design rests on three ideas. Test at the cheapest level that can catch the bug: kernels on simulators and CPUs, a handful of small models on hardware for each change, the full model set nightly, and long runs only before a release.

Compare against a trusted reference with the right tolerance, never bitwise: floating point results change with summation order, so correctness means agreeing with a golden result within a stated tolerance, and for training, following the reference loss curve and reaching the reference eval score. Make every result attributable: each run records the exact versions of all layers and the node it ran on, so a regression can be bisected to one change and routed to one team, and a release is a pinned bundle in Git that can be reproduced or reverted.

Scope it first

Which layers change and how often? — Model code, framework, compiler and kernels many times a day; runtime and driver daily; firmware weekly; hardware revision every year or two
How many changes a day? — About 400 merged changes across all layers, about 240 of which touch the hardware path
How many models must keep working? — 60 models in the qualification set, 6 of them large models that need many nodes
How much accelerator capacity does CI get? — 16 nodes of 8 chips (128 chips) across two hardware revisions; developers' experiments share the same pool
Training, inference or both? — Both; inference is what the production fleet serves, training runs on it too
What is a release? — A tested tuple of compiler, kernels, runtime, driver and firmware versions, plus the model serving configs, pinned in Git
What may never happen? — A release that makes a production model less accurate than its accepted reference, or that cannot be rolled back

The interviewer listens for four things. Do you match each test tier to the cheapest place that can run it, and say what runs per change, nightly and per release? Do you define correctness as agreement within a tolerance against a trusted reference, with loss-curve and eval checks for training, instead of bitwise equality? Do you treat performance numbers as noisy samples that need repeats, warmup and a statistical test before you call a regression and bisect it? And do you spend scarce accelerator time deliberately, with queues, priorities, preemption and impact-based selection?

Key concepts

Compatibility matrix and release bundle
Test tiers by cost
Golden references and tolerances
bf16 and fp8 numerics
Loss-curve and eval gates
Warmup and run-to-run noise
Mann-Whitney and bootstrap
Bisect to the culprit
Hardware health triage
Queues, priorities and preemption

Functional requirements

Build every change into a versioned stack bundle (compiler, kernels, runtime, driver, firmware) and record those versions on every test result
Run kernel and compiler tests on simulators and CPUs for every change, and a small-model smoke test on hardware for every change that touches the hardware path
Run the full qualification set of 60 models nightly for correctness and performance, and a longer qualification before every release
Detect correctness and performance regressions automatically, bisect them to a change, and open a ticket for the owning team
Quarantine unhealthy accelerator nodes and rerun the tests they failed on healthy nodes
Keep the desired stack and model configs of every environment in Git; promote by pull request and roll back by revert
Share the accelerator pool between presubmit, nightly, release, bisect and developer experiments by priority and quota

Non-functional requirements

Time to trustworthy signal: presubmit hardware result within 45 minutes at the 90th percentile; nightly report before the working day starts (our targets)
Correctness: no release reduces a production model's eval score below its accepted threshold (our gate, for example 99% of the reference score)
Performance: a sustained throughput drop of 2% or more on any qualification model is detected within one nightly run (our target)
Flake rate of hardware tests below 1% of runs, with every flaky test owned and tracked (our target)
Reproducible: any past result can be rerun from its recorded bundle, config and seed
Auditable: every change to an environment is a reviewed commit with the gate results attached

Interview questions

How do you decide whether a model is still correct after a compiler change?
Your nightly shows llm-70b 3% slower. What happens next?
How many benchmark runs do you need, and why not just one?
How do you keep the test matrix affordable?
How do you tell a hardware fault from a software regression?
How does GitOps apply to an accelerator software stack?
How do you share 128 chips between CI and developers?
Which numbers tell you the CI itself is healthy?