Performance Baselines, Regression Detection & Auto-bisect for Accelerators

Catching model performance regressions on accelerators: benchmark hygiene (warmup, pinned clocks, isolation, repeats), the metrics that matter (tokens/s, TTFT, step time, MFU, percentiles, memory), noise and confidence intervals, change-point detection with E-Divisive and CUSUM, baseline management, auto-bisect across many components, and routing each regression to an owner.

Overview

The job

On an accelerator program, performance is the product. A 3 percent drop in training throughput on a frontier run is weeks of cluster time; a 10 percent rise in time to first token is felt by every user. Every day brings compiler, kernel, runtime and firmware changes, and benchmark results move by a percent or two on their own.

The performance CI owner builds the measurement system that separates signal from noise automatically, every night, across hundreds of model configurations, then finds the change behind a real regression and puts it in front of the team that owns it, with evidence, before the next release.

Interview questions

  • Design the performance regression system for 300 model benchmarks that run nightly on accelerators.
  • Run-to-run noise on a benchmark is 3 percent. The team wants to catch 2 percent regressions. What do you do?
  • Compare CUSUM and E-Divisive for performance CI. When would you use each?
  • A nightly regressed tokens/s by 4 percent. Since the last good night: 120 compiler commits, 30 kernel commits, a runtime minor version and a firmware update. Find the culprit.
  • What do you control and record to make an accelerator benchmark trustworthy?
  • How do you manage baselines across hardware generations and intentional regressions?

Sign up free to read the full topic

More AI Infrastructure topics

Start free on Cariara · Coding problems · System design studio