Model CI on Custom AI Accelerators: Strategy, Test Matrices & Release Qualification

Continuous integration where the unit under test is a frontier model on in-house silicon: test tiers from per-commit to release candidate, the models x configs x hardware x software x environment matrix and how to prune it, functional, performance and stability suites, and turning each release risk into an automated gate with an owner.

Overview

The job

A company that builds its own AI accelerator (the way Google ships TPUs, AWS ships Trainium and Meta ships MTIA) ships a whole stack with the chip: a graph compiler, kernels, a runtime, collectives, firmware, a driver, and the boards and racks underneath. Model teams ship new architectures every few weeks, and every layer merges changes every day. Model CI answers one question continuously: do the models the company depends on still train and serve correctly, fast, and without falling over, on every generation of the chip in the fleet?

The engineer who owns it does not write the compiler. They own the strategy (what runs where and how often, inside a chip-hour budget), the test matrix, the functional, performance, stability and release-qualification pipelines, and the risk register that turns each partner team's worry into a gate that runs on its own and pages an owner.

Interview questions

  • You join a new accelerator program as the owner of model CI. Silicon arrives in six months. What do you build, in what order?
  • The matrix is 15 models x 12 configs x 3 hardware generations x 8 software combinations x 4 environments, and the nightly budget is about 2,000 chip-hours. How do you choose what runs?
  • How do you turn release risks from the compiler, kernel, runtime, firmware and hardware teams into automated gates?
  • Design stability testing for a training stack: how long, how many chips, and what counts as a failure?
  • The day before a planned release, one blocking gate fails: tokens/s on the MoE model is 4 percent below baseline. Everything else is green. What do you do?
  • How do you run meaningful model CI before silicon exists, and how do you switch over when it arrives?

Sign up free to read the full topic

More AI Infrastructure topics

Start free on Cariara · Coding problems · System design studio