Ephemeral CI Runners on AWS

Federate GitHub Actions to AWS with OIDC (no static keys) and run ephemeral EC2 self-hosted runners with least-privilege IAM, private-subnet VPC design, and spot handling.

Overview

Why Ephemeral CI Runners on AWS

GitHub-hosted runners are convenient but give you no control over the network they run in, the IAM identity they carry, or the hardware they use. For a CI/CD stack that builds and tests against AWS resources (private artifact mirrors, ECR, GPU AMIs), you want runners that live inside your own VPC, assume a scoped IAM role, and disappear after a single job so no state leaks between builds.

The modern pattern has two independent halves. First, federate identity: GitHub Actions mints an OIDC token per job, and AWS trades it for temporary credentials via sts:AssumeRoleWithWebIdentity. This eliminates every long-lived AWS_ACCESS_KEY_ID secret from your repo. Second, run the job on an ephemeral EC2 instance: a webhook fires on workflow_job queued, a Lambda scales up an instance from a pre-baked AMI, the instance registers itself as a just-in-time (JIT) runner with --ephemeral, executes exactly one job, and terminates.

The Two Halves Are Orthogonal

OIDC federation governs what the workflow can do inside AWS from the GitHub-hosted control plane or from a runner. The ephemeral EC2 fleet governs where the job physically executes. You can adopt OIDC without self-hosted runners, and you can run self-hosted runners that still use OIDC for cloud access. This topic covers wiring both, plus the VPC, security-group, and cost decisions that make a private ephemeral fleet actually work.

Reference Implementation

The philips-labs/terraform-aws-github-runner Terraform module is the de facto reference: webhook -> API Gateway -> Lambda -> SQS -> scale-up Lambda -> EC2. You do not have to use it, but interviewers expect you to reason about that exact control flow and where each AWS primitive fits.

Key concepts

OIDC federation (AssumeRoleWithWebIdentity): GitHub Actions exposes an OIDC provider at https://token.actions.githubusercontent.com. A job requests a signed JWT, and AWS STS validates it against a registered IAM OIDC identity provider, then returns short-lived credentials via sts:AssumeRoleWithWebIdentity. No static keys are stored anywhere.
Trust policy sub and aud conditions: The IAM role trust policy pins the token's audience with token.actions.githubusercontent.com:aud equal to sts.amazonaws.com and scopes the subject with token.actions.githubusercontent.com:sub (for example repo:my-org/my-repo:ref:refs/heads/main or repo:my-org/my-repo:environment:prod). A missing or wildcard sub lets any repo assume the role.
Ephemeral JIT runner: A runner registered with a just-in-time config token and started with --ephemeral accepts exactly one job, then deregisters. Combined with instance termination this guarantees a fresh host per job, so no build state, secret, or cache survives into the next job.
Scale-up / scale-down control flow: The philips-labs/terraform-aws-github-runner pattern: a workflow_job webhook hits API Gateway, a Lambda validates the HMAC signature and enqueues to SQS, a scale-up Lambda launches an EC2 instance, and a scheduled scale-down Lambda reaps idle or orphaned instances. SQS decouples burst webhook traffic from EC2 launch throttling.
Custom runner AMI: A machine image built with packer (or EC2 Image Builder) that pre-bakes the runner agent, Docker, language toolchains, and GPU drivers. Baking cuts per-job cold start from minutes to seconds and pins a tested, immutable toolchain versioned by AMI ID.
Runner instance role vs deploy role: Two distinct IAM identities. The EC2 instance profile role grants only what the host needs to bootstrap (fetch the registration token from SSM, pull the AMI, write logs). The workflow's OIDC-assumed role grants the deploy or build permissions. Never merge them, or every job inherits the host's bootstrap rights.
VPC endpoints / PrivateLink: Interface and gateway VPC endpoints (com.amazonaws..ecr.dkr, .ecr.api, .s3, .sts, .logs, .ssm) let runners in private subnets reach AWS APIs without a NAT gateway or public IP, cutting NAT data-processing cost and keeping traffic on the AWS backbone.

When to use it

Your CI jobs need to reach private AWS resources (RDS, internal ECR, VPC-only artifact mirrors) that GitHub-hosted runners on public IPs cannot touch.
You are eliminating long-lived AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY repo secrets and want short-lived, per-job credentials via OIDC.
Builds require specialized or GPU hardware (for example AMD GPU CI) or large instance types that GitHub-hosted runners do not offer.
You need each job to run on a clean, isolated host so secrets, caches, or Docker layers never leak between untrusted builds.
Build volume is spiky and you want to scale runner capacity to zero between jobs and ride spot pricing instead of paying for idle always-on hosts.

Interview questions

How do you set up the GitHub-to-AWS OIDC trust so workflows get AWS credentials with no static keys?
Design the ephemeral EC2 runner lifecycle end to end. What launches an instance and what tears it down?
How do you lock down the IAM role attached to the runner EC2 instance?
Runners sit in private subnets and must reach an internal artifact mirror with no public IP. How do you wire egress?
You run on spot to cut cost. How do you survive an interruption mid-build?
How do you bake a runner AMI with the toolchain pre-installed and roll it out safely?