Hardware-in-the-Loop CI
How to wire real GPUs and boards into CI as hardware-in-the-loop test benches: self-hosted runners, exclusive leasing, flash-test-recover loops, and bricked-board recovery.
Key concepts
- Hardware-in-the-Loop (HIL)
- A CI model where the test target is real physical hardware wired into the pipeline, so a job flashes, boots, and exercises an actual GPU or board rather than an emulator. Used when the failure mode lives in silicon, firmware, or the driver-to-hardware boundary and cannot be reproduced in software.
- Bench node and self-hosted runner
- A bench is a host machine with one or more real GPUs installed plus power control and serial capture. Each bench runs a self-hosted GitHub Actions runner (the run.sh agent registered with config.sh) so jobs targeting that hardware land on that specific machine. One runner process typically owns one board's worth of exclusive access.
- Runner labels and targeting
- Custom labels attached at registration (--labels self-hosted,gpu,mi300x,vbios-flashable) let a workflow's runs-on array select the exact product and capability it needs. Labels are the routing layer between an abstract job and the physical part, so runs-on: [self-hosted, mi300x] only ever executes on a bench holding that ASIC.
- Exclusive lease and serialization
- Because a physical board cannot be time-sliced like a VM, only one job may touch it at a time. A lease is a mutual-exclusion primitive (a flock file lock, a GitHub concurrency group, or an external reservation service) that grants one job sole ownership of a board for its duration and blocks or queues everyone else.
- Flash-test-recover loop
- The core HIL cycle for a flashable target: reflash the firmware or VBIOS, power-cycle the part via PDU or ipmitool so the new image is loaded cold, run the validation suite while capturing the serial console, then restore a known-good state. Every stage must assume the previous one can leave the board wedged.
- Out-of-band management (BMC/IPMI/PDU)
- A control path independent of the host OS and the device under test: a baseboard management controller reachable over ipmitool, a network-controlled PDU driven by pdudaemon or SNMP, and a serial concentrator via conserver. Out-of-band control is what lets CI hard power-cycle and recover a board even when the OS or the GPU is fully hung.
- Golden recovery image and watchdog
- A known-good firmware or VBIOS kept on a secondary flash bank or recovery jumper, plus a hardware or software watchdog that detects a board that failed to enumerate after boot. Together they let an automated recovery step reflash the golden image and reclaim a bench that a bad experimental flash would otherwise turn into a dead runner.
When to use it
- A change touches firmware, VBIOS, the amdgpu kernel driver, or low-level ROCm runtime code where only real silicon can reproduce the fault, and an emulator or ubuntu-latest VM cannot exercise the PCIe or SPI flash path.
- You need pre-merge validation on a specific ASIC or board revision (for example mi300x versus gfx1100) that must run on labeled hardware rather than any generic runner.
- You are validating a flashable artifact end to end: reflash the part, power-cycle it, boot, and confirm the device enumerates and passes compute tests before allowing merge.
- Bug reproduction requires deterministic control of the physical environment such as power cycling, thermal state, or clean cold boots that a shared VM cannot provide.
- You have a limited pool of expensive boards shared across many concurrent PRs and need leasing, queueing, and fair scheduling so jobs do not collide on the same physical part.
Interview questions
- How do you wire a GPU test bench into GitHub Actions as a self-hosted runner that only accepts jobs for its specific hardware?
- How do you serialize exclusive access to a limited pool of boards so many concurrent jobs never collide on the same physical part?
- How do you automatically recover a board that a bad firmware flash just bricked, so you do not permanently lose the runner?
- How do you capture and report HIL results and serial console logs back to the PR?
- How do you scale a device farm when you have far more PRs than physical boards?
- How do you reduce flaky hardware failures so real regressions are not lost in physical-hardware noise?
Sign up free to read the full topic
More DevOps topics
Start free on Cariara · Coding problems · System design studio