Self-Hosted CI Runners at Scale

Provision, autoscale, and secure self-hosted GitHub Actions runners for GPU/firmware CI using ephemeral JIT runners, runner groups, and actions-runner-controller.

Overview

Why self-hosted runners

GitHub-hosted runners give you clean, disposable Ubuntu, Windows and macOS VMs. They cannot touch your internal network, cannot mount your NFS artifact stores, have no GPUs, and reset every job. A firmware, kernel or ROCm CI pipeline needs the opposite.

Internal network and NFS — No — Yes
AMD GPUs for hip/ROCm — No — Yes
Hardware signing keys or HSM — No — Yes, on a restricted class
Clean machine every job — Yes — Only if ephemeral
Capability routing, for example firmware-build — No — Yes, by labels

How a runner gets work

Self-hosted runners are agents on your own hardware or cloud. They poll GitHub over an outbound HTTPS long-poll, so no inbound ports are open. A runner registers against a repo, an org or an enterprise, advertises labels, and picks up any queued job whose runs-on matches.

> [!IMPORTANT] The design problem: keep long-lived, network-privileged, secret-adjacent machines from becoming the softest attack surface in your supply chain, while still autoscaling them for bursty kernel and ROCm builds.

The core tension

A persistent runner is convenient: warm caches and pre-pulled containers. It is also dangerous, because state from one job leaks into the next and a malicious PR can poison the workspace for every later build.

Persistent — Warm caches, pre-pulled containers — Workspace poisoning, credential and state leaks
Ephemeral JIT via actions-runner-controller (ARC) — Clean machine per job, autoscaled — Cold starts

This topic covers the ephemeral architecture end to end:

Registration — Repo, org, enterprise scope
Routing — runner groups and labels
Lifecycle — JIT and ephemeral runners
Scaling — ARC autoscaling
Hardening — Images and the security model
Isolation — Routing a privileged firmware-signing build

Key concepts

Registration scope (repo / org / enterprise): A runner is registered against one scope using a short-lived registration token from the GitHub API. Repo runners (/repos/{owner}/{repo}/actions/runners/registration-token) serve one repo; org runners serve any repo in the org and are routed via runner groups; enterprise runners span multiple orgs. Prefer org or enterprise scope so you manage one fleet, not one pool per repo.
Runner groups: An org/enterprise-level access boundary that controls which repositories may send jobs to a set of runners. You place GPU or signing runners in a restricted group (for example firmware-signers) and allow only specific repos, and you can forbid public repositories entirely. Groups govern access; labels govern matching.
Labels and runs-on matching: Each runner advertises labels (defaults like self-hosted, linux, x64, plus custom ones like rocm, gpu-mi300, nfs, signing). A job's runs-on must match every label in its array, so runs-on: [self-hosted, linux, rocm, nfs] only lands on a box carrying all four. Labels are how you route heterogeneous hardware.
Ephemeral / just-in-time (JIT) runner: A runner started with --ephemeral (or provisioned via the JIT config API) that accepts exactly one job, then deregisters and exits. JIT config returns an encoded one-shot registration so no registration token or .runner credential persists on disk. This guarantees a clean machine per job and eliminates cross-job state leakage.
actions-runner-controller (ARC): A Kubernetes operator that runs the fleet as pods. The current model uses gha-runner-scale-set Helm charts: a listener authenticates to GitHub, watches for queued jobs matching the scale set's runnerScaleSetName, and scales ephemeral runner pods between minRunners and maxRunners. Each pod is one ephemeral runner.
Queue-depth (event-driven) autoscaling: Scaling driven by actual demand rather than CPU metrics. ARC's listener receives the count of assigned/queued jobs for its scale set from GitHub and creates exactly that many runner pods (capped by maxRunners). The legacy pattern used workflow_job webhooks into a HorizontalRunnerAutoscaler; both scale on job queue depth, not host load.
Hardened runner image: A minimal, pinned, non-root container or VM image baked with only the required toolchain (ROCm, cross-compilers, signing client), a pinned runner version, no embedded secrets, egress-restricted networking, and read-only root where possible. Secrets and NFS/HSM access are injected at runtime via short-lived credentials, never baked into the image.

When to use it

You need hardware GitHub does not offer — AMD GPUs for ROCm/HIP tests, large-memory kernel build boxes, or specific CPU microarchitectures — so hosted runners are a non-starter.
Builds must reach internal-only resources: NFS-mounted toolchains and sccache/ccache stores, an internal artifact registry, or a firmware signing service reachable only from a trusted VLAN.
You run bursty, heterogeneous workloads (nightly kernel matrices, per-PR ROCm builds) and need queue-depth autoscaling instead of a fixed pool that is either idle or saturated.
Supply-chain security requires one job per machine with no state carryover, no long-lived secrets on disk, and isolation of privileged signing steps from untrusted PR code.
Cost or data-residency rules mean CI compute must stay in your own cloud account or on-prem datacenter rather than on GitHub-hosted infrastructure.

Interview questions

How do you stand up a fleet of ephemeral just-in-time (JIT) runners so each build gets a clean machine?
How do you autoscale runners on Kubernetes with actions-runner-controller (ARC) based on queue depth?
How do you secure a self-hosted runner fleet against supply-chain and cross-job attacks?
A runner registers and shows online in the UI but never picks up any jobs. How do you debug it?
How do you size and route jobs across a heterogeneous fleet (GPU boxes, big-memory kernel builders, generic runners)?
How do you isolate a privileged firmware-signing build so untrusted code can never reach the signing key?