Running CI on Scarce Accelerators: Scheduling, Artifacts & Cost

The CI platform under accelerator testing: pools by hardware generation, Kueue, Slurm and self-hosted runners, fair share and preemption between teams, health checks and quarantine of bad nodes, artifact and cache management for weights and compiled binaries, CI observability and cost per signal.

Overview

The job

CI on CPUs treats machines as free and interchangeable: add runners until the queue is short. CI for a model stack on custom accelerators cannot. The accelerators are the scarcest, most expensive hardware the company owns, several generations run side by side, one test may need a full node or a multi-node slice, and a node can fail in ways that look exactly like a software bug.

Picture the compiler and runtime org at a company building its own AI accelerator: 600 accelerator nodes across three generations, eight teams sharing them, and a merge queue that must give each pull request a signal within the hour. The CI infrastructure engineer owns the layer under the tests: which job runs where and in what order, how teams share the pools, how bad hardware is kept away from results, where weights and compiled artifacts live, and what one useful signal costs.

Interview questions

  • Design the CI platform for 600 accelerator nodes across three hardware generations, shared by eight teams.
  • Presubmit queue times doubled this week. How do you find out why?
  • A test fails intermittently on accelerator CI. How do you decide whether the hardware, the platform or the code is at fault?
  • Model weights for integration tests total 40 TB and every job downloads its own copy. Fix it.
  • Leadership asks you to cut accelerator CI cost by 30% without slowing developers down. What do you do?

Sign up free to read the full topic

More AI Infrastructure topics

Start free on Cariara · Coding problems · System design studio