CI/CD for Server Firmware: OpenBMC, Zephyr and EDK2

Builds, caches, board farms, signing and releases for many silicon steppings

Design the CI/CD platform that builds, tests on real boards, signs and releases server firmware (OpenBMC, Zephyr and EDK2) for dozens of teams, many silicon steppings and board variants, with code in GitLab and Perforce.

Overview

A company designs server SoCs. Dozens of teams write the firmware that ships with them: the BMC (baseboard management controller) firmware built from OpenBMC, a Linux distribution built with Yocto and BitBake; small microcontroller images built on the Zephyr RTOS with west and CMake; and the host UEFI firmware that boots the main processors, built from EDK2. Most code is in GitLab; some teams, often those holding silicon vendor drops and large binaries, still work in Perforce. Every chip comes in several steppings (A0, B0, B1: revisions of the silicon), and each stepping runs on several board variants (reference boards and customer platforms).

You are asked to design the CI/CD platform for all of it: how code is built, cached, checked and tested on real hardware, how images are signed, and how releases are cut and traced. The interviewer is not asking how to write one Jenkinsfile. They want to hear how you keep a Yocto build from taking three hours on every change, how you share 54 boards among 600 engineers, how you tell a flaky board from a flaky test, how one fix reaches five product branches, and where the signing keys live.

Three sibling topics cover what this one does not repeat. Platform Provisioning on Bare Metal: From Silicon to Kubernetes covers rolling finished firmware out to a running fleet. CI/CD and GitOps for a Multi-Cloud and On-Premises Fleet covers services and clusters. Pre-Silicon Software: Virtual Platforms and SystemC TLM in CI, CI for Large Models on Custom AI Accelerators, and Monorepo Builds at Scale: Dependency Graphs, Test Selection and Change Impact each cover their own subject; here they are named only where they connect.

Questions to ask first

How many people and teams? — 600 engineers in 40 teams, across three firmware stacks
How much change? — About 500 change pushes a working day, twice the average at the peak hour; about 300 merges a day
Which silicon and boards? — 2 SoC generations in production and 1 in bring-up: 6 live steppings, 24 board variants
What does one board need? — One BMC image, one host UEFI image and two microcontroller images
Where is the code? — GitLab for most repositories; Perforce for vendor drops, large binaries and two legacy teams
What must a merge prove? — It builds for every affected board, passes static analysis and unit tests, and boots on a real board of the main stepping
Who may sign production images? — Only a release pipeline, through a signing service, with keys in an HSM and two-person approval
Is there a safety or compliance standard? — MISRA C for the microcontroller code; requirements tracked in Jira

Building blocks and tool choices

Firmware runs on your own hardware, so there is no AWS, Azure or Google Cloud equivalent for the boards themselves. The table maps each building block to an open source choice, a commercial one, and where cloud-hosted runners or storage fit.

CI orchestrator — Jenkins; GitLab CE; Gerrit with Jenkins (the OpenBMC project's own setup) — GitLab Premium or Ultimate, GitHub Enterprise, CloudBees CI — GitLab.com or GitHub.com, with self-managed runners for boards
Runners — Jenkins Kubernetes plugin, GitLab Kubernetes executor, Actions Runner Controller — Same, with vendor support — GitLab Docker Autoscaler on AWS, Azure or Google Cloud VMs (fleeting plugins)
Build caches — sstate over HTTP or NFS, bitbake-hashserv, own-mirrors — An artifact manager — Object storage (S3, Blob Storage, Cloud Storage) behind HTTP
Source control — Git, Gerrit, git-p4, Git LFS — Perforce Helix Core — Hosted GitLab or GitHub
Static analysis — clang-tidy, cppcheck with its MISRA addon, CodeChecker — Coverity, Polyspace, ECLAIR, Parasoft C/C++test — Runs on the same runners
Coverage — gcov with gcovr or lcov — Vendor on-target coverage tools — Host and QEMU coverage on cloud runners
Board farm — labgrid, LAVA, PDUDaemon, OpenOCD, flashrom — Lab automation from test equipment vendors — None: unreleased silicon stays on premises
Signing — An internal service using PKCS#11 to an HSM — Network HSMs, code signing products — Cloud HSM services, if policy allows keys there
Farm configuration — Ansible, Chef — Ansible Automation Platform, Chef commercial — None
Traceability — Commit trailers, SPDX SBOMs — Jira — Jira Cloud

The interviewer listens for four things. Do you treat each firmware stack as its own pipeline with its own cache, rather than one generic build? Do you size the board farm as a queue with a wait-time target? Do you separate test failures from infrastructure failures before anyone is paged? And do signing keys stay inside an HSM, with build agents only ever sending digests?

What you will be able to do

Lay out pipelines for OpenBMC, Zephyr and EDK2 on ephemeral runners, and size the runner pool from change volume.
Explain Yocto shared state, hash equivalence and download mirrors, and show how cache hit rate drives build time.
Design a hardware-in-the-loop farm with power control, consoles, flashing and recovery, and size each board class from a wait-time target.
Tell a flaky board from a flaky test automatically, and quarantine either without blocking merges.
Sign images with keys that never leave an HSM, and trace a released image back to commits, test runs and Jira requirements.

Key concepts

Pipeline per firmware stack
Ephemeral runners
Yocto sstate and hash equivalence
Download mirrors
West manifests and Twister
Stuart CI for EDK2
Git and Perforce coexistence
Release branches per product
Hardware-in-the-loop farm
HSM-backed signing service

Functional requirements

Build every affected board variant for each change pushed to GitLab or submitted in Perforce, for all three firmware stacks
Run unit tests, static analysis (including MISRA C checks for microcontroller code) and coverage on every change
Boot and smoke-test each merge candidate on a real board of the main stepping before it merges
Run a nightly full regression on every board variant and stepping, and publish results per board
Schedule scarce boards fairly with priorities, reservations for debugging, and automatic recovery of bricked boards
Sign release images (BMC, UEFI capsules, microcontroller images) through a signing service that holds keys in an HSM
Cut releases from release branches, attach an SBOM and test evidence, and link each change to its Jira requirement
Keep the farm's hosts, PDUs, programmers and board inventory under configuration management

Non-functional requirements

Pre-merge feedback for a typical change within 45 minutes at the 90th percentile (a target we chose)
Merge smoke tests wait no more than 10 minutes for a board at the 95th percentile (a target we chose)
Reproducible: a release can be rebuilt bit for bit from its pinned manifests and mirrors, even if upstream sites vanish
No signing key on any build agent; production signing needs two approvers and leaves an audit record
Infrastructure failures (a dead board, a full disk) never show to a developer as a test failure
A board class losing half its boards degrades merge latency, not correctness
Release branches stay buildable for the support life of the product, which outlives several upstream releases

Interview questions

Walk me through what happens when an engineer pushes an OpenBMC change.
Why does a small change rebuild so much in Yocto, and how do you fix it?
How many boards do you need, and how do you decide?
A hardware test fails. How do you know if it is the change, the test or the board?
How do you handle many steppings and board variants without drowning in branches?
Where do the signing keys live, and how does a build get a signature?
How do GitLab and Perforce work together?
How do you make a release rebuildable in five years?
How do you show that a released image meets its requirements?